jFed implements support the Experiment Specification format (“ESpec”).

This page defines the specification for the format.

General

The Experiment Specification format is not a replacement for the RSpec format. An ESpec contains an RSpec and combines it with other files.

The purpose of an RSpec is to define which resources are needed.

The purpose of an ESpec is to additionally define which files should be placed where, and which scripts should be started.

The current ESpec specification already allows some complex ESpecs. However, the base idea when using an ESpec should be to keep it simple, and to put as little as possible in the ESpec. It is meant for easily bootstrapping your experiment. It is not meant for running experiment logic.

It is preferable to keep ESpecs so simple that a user not familiar with the format, will be easily able to manually execute your experiment using tools that do not support ESpec.

ESpec bundles

An ESpec bundle is a group of files, which contains:

  • experiment-specification.yml which contains the meta data that describes what to do with the other files
  • An RSpec
  • Zero, one or more files to upload
  • Zero, one or more scripts to execute

There are different ways to “bundle” the files that form an ESpec:

  • Place them in a single directory
  • Place them in an archive file (.zip, .tar, .tar.gz, .jar, …)
  • Place them in a git repo
  • Place them in a github repo

Currently jFed supports all these methods. The git and github methods are currently not yet supported in the Experimenter GUI, but are supported in the automated tester.

Format of experiment-specification.yml

The ESpec meta data file, experiment-specification.yml uses YAML syntax (short tutorial).

A basic file looks like this:

version: 1.0-basic
rspec: nodes.rspec
upload: exp-data-files.tar.gz
execute: exp-script.sh

Version should for now always be “1.0-basic”. Future versions will use another identifier.

Note that execute will first act as an upload, and then also execute the uploaded file.

Both upload and execute allow multiple files to be specified. Use a list for that. Example:

version: 1.0-basic
rspec: nodes.rspec
upload: 
   - exp-files-set1.tar.gz
   - exp-files-set2.zip
execute:
   - setup.sh
   - exp-run.sh

Files are uploaded in parallell, but scripts are always executed in order. So in this case, exp-run.sh will not run before setup.sh has run. (todo decide if this synchronizes over multiple nodes by default)

Each time a filename is specified, it is assumed it refers to a bundled file. You can also directly specify the content of the file, or provide an URL to download it from. To do that, the “long” format is used. This also allows extra options such as to which node, and in which path to upload the file.

An example:

version: 1.0-basic
rspec:
   - bundled: 3-nodes.rspec
upload:
   - exp-files-set1.tar.gz
   - bundled: exp-files-set2.tar.gz
     path: /tmp
     nodes: [central, exp1]
   - download: http://example.com/exp-files-set3.tar.gz
   - direct: |
        You can also directly specify the content of a file. This text will thus be stored on all nodes in /tmp/demo.txt
        Check the yaml syntax of "literal-blocks" for details about syntax and removing indentation
     path: /tmp/demo.txt
execute:
   - bundled: setup-central-node.sh
     nodes: central
   - bundled: setup-exp-node.sh
     nodes: [exp1, exp2]
   - local: /work/repo/start-exp.sh
     nodes: [exp1, exp2]

If not path is specified, the home dir of the user is used. This default can be changed by using dir. dir will also create the directories if needed. While this feature can be convenient, keep in mind that permissions of the logged in user are used. So directory creation is typically not allowed everywhere.

You can specify the destination dirs for uploads and scripts (where files in execute are placed) seperately. If scripts is not specified, it defaults to the same dir as uploads. If uploads is not specified, it defaults to the users home dir.

Paths may start with ~ to indicate they are relative to the users home dir.

An example:

version: 1.0-basic
rspec:
   - bundled: example.rspec
dir: 
   - path: /tmp/data/
     content: uploads
   - path: ~/scripts/
     content: scripts
   - path: /tmp/extra/
upload:
   - exp-files-set1.tar.gz
   - exp-files-set2.tar.gz
   - bundled: extra.tgz
     path: /tmp/extra/
execute:
   - setup.sh
   - start.sh

In this example, if the users home dir is at /home/someuser/ the files will be places in:

  • /tmp/extra/extra.tgz
  • /tmp/data/exp-files-set1.tar.gz
  • /tmp/data/exp-files-set2.tar.gz
  • /home/someuser/scripts/setup.sh
  • /home/someuser/scripts/start.sh

Dir details

Each dir entry supports the following options:

  • path (string) MANDATORY
  • content (string)
  • permissions (string)
  • nodes (empty, string, or list of string)
  • sudo (boolean)

path specifies the path of the directory used in the ExperimentSpecification. This needs to be an absolute path, or a path relative to the user homedir. Thus it needs to start with either / or ~.

Non existing directories will be created (including parent directories). Existing directories will be left as is (unless their permissions are wrong, see later).

content is either upload, scripts or ansible, or left unspecified. If specified, files in the referenced sections, for which no path or a relative path is given, will be stored in (or relative to) this dir. If content is unspecified, the dir will just be created if it doesn’t exist, which is often usefull on its own.

permissions are the required permissions for the directory. If not specified, a default value of u=rwx is used. The formats supported by chmod are supported, so octal notation (0600) and symbolic notation (uog=rwx).

nodes is a list empty or unspecified, or one or more nodes on which the dir is created and used as specified in content. If empty, all nodes are used (except for the ansible control machine).

sudo if not specified, this option is false. If true, the directory will be created by using sudo to gain root priviledges. This can be required to create icertain directories. Off course, this requires the nodes to have a correctly configured sudo command.

File content details

Both upload and execute need a list with zero, one or more objects that specify what to do. In both cases, these objects contain at least information on the content of the file that is worked with.

To specify file content, the object contains a specific key-value pair. The key determines the method to retrieve the file content, the value depends on the key but typically specifies which specific content to fetch using the method specified in the key. The value is typically a string, but can be an object for certain keys, if they require additional data.

There are different ways to specify which file content to use:

  • bundled: The file is bundled in the ESpec bundle. The value is the name of the bundles file, possibly including the relative path inside the ESpec bundle. You can also specify a directory from the bundle.
  • download: The file must be downloaded. The value is the URL of the file.
  • git or github: The file (or dir) is in a git repository. There is currently no difference between specifying git or github. The value is a string with the git URL of the public repo. It is also possible to specify more options, using an object as value instead. The following fields are then supported:
    • url: The git URL of the repo. May be an HTTP or SSH git url. (mandatory)
    • branch: The branch of the repository to use. (optional, default is “master”)
    • dir: The subdir in the repo to use. If file is specified, this is the base dir of the file. (optional)
    • file: The file to fetch from the git repo. If this is not specified, an entire dir, or the entire repo is used. (optional)
    • username: The username used for basic authentication. (optional, may not be combined with privateKey)
    • password: The password matching the username used for basic authentication. (mandatory if username is specified, forbidden otherwise.)
    • privateKey: The private key (in PEM format) needed to access this git repo. Note that the username is specified in the git URL in this case. (optional, may not be combined with username or password)
  • meta: The file is generated by the client, and contains meta data about the experiment/slice. The value is the required meta data. There is currently 1 supported value:
    • manifest.xml: The manifest RSpec of the slice. If there are multiple AM’s involved in this slice then this is the combination of all of their manifest RSpecs.
    • experiment-info.json: Information about the experiment in JSON format. This includes info on the user, project, slice, ssh users, and on the nodes.
    • client_id.txt: The client_id of the node.
  • generate: The file must be generated by the client. The value is an object or string. The object should always contain a method field. The string is shorthand for an object with only the method field, with as value the string. There are currently 2 supported methods:
    • keypair: This method requires no extra fields. It will generate a random keypair, pass that keypair in the Provision phase, and upload the keypair. This way, all nodes in the experiment can afterwards securely communicate over SSH.
    • random: Generate a file with random content. There are 2 extra fields needed: format and length.
      • format: Specifies which form the random data has. Options are: password, binary, base64 and alphanum
      • length: The length in bytes of the generated random data. Note that for base64 this is the length of the encoded bytes, not the length of the resulting base64 string.
    • rspec: Generate am RSpec file. There are 1 extra fields needed: am, and there are some optional fields nodes, prefix and icon.
      • am: The component_manager_id of the nodes in the RSpec. You may specify either the URN, the server ID (an integer), or the testbed ID (a string).
      • nodes: Either the amount of nodes that need to be generated (an integer), or a list of names for the nodes (a list of strings). Default is 1 (= generate 1 node).
      • prefix: The prefix used to generate node names, if the nodes option is an integer. The default is “n”. (example: For nodes: 2 this would cause 2 nodes to be generated, with names “n1” and “n2”.)
      • icon: The “icon” in the jFed Experimenter GUI to use. If not specified, an icon will be chosen automatically (which is almost always what you need). The name of this icon is the ResourceClass ID that can be found in the fls-api. Examples are: physical-node, wireless, generic-node, vm-xen, vm-openvz, vm, lte, docker-container. For the full list, check the API

And example with generated content:

version: 1.0-basic
rspec:
   generated:
       method: rspec
       am: iminds-docker
       nodes:
         - client
         - server
upload:
   - generated: keypair
   - generated:
      method: random
      format: password
      length: 20
     path: random-password.txt
   - generated:
      method: random
      format: binary
      length: 500
     path: /tmp/seed-data.dat
   - git: user@git.example.com:/example-experiment/my-experiment.git
     path: ~/my-exp-repo/
execute:
   - direct: |
       #!/bin/bash -xe
       
       ~/my-exp-repo/prepare-experiment.sh --set-password ~/random-password.txt
       
       ~/my-exp-repo/run-experiment.sh --password ~/random-password.txt --seedfile /tmp/seed-data.dat

       echo 'All done'

For execute, playbook and galaxy, it is allowed not to specify a file content source, but to only specify a path.

In this case, it is assumed the the file already exists om the remote node’s filesystem at the given path. The file is thus not uploaded to the node, but is already present somehow. Note that is up to you to make sure the file exists. If you want, the file can have different content on each node it is used on.

There are multiple reasons why a file would already be present: it can be included in the diskimage the nodes runs, it can be installed using the install service of the RSpec, or a previous upload or execute step in the ExperimentSpecification can have created it (directly or indirectly).

An (contrived) example:

version: 1.0-basic
rspec: experiment.rspec
upload:
   - bundled: check.sh
     permission: u=rx
execute:
   - path: check.sh
   - path: /usr/bin/sync
ansible:
    host:
       upload:
             - bundled: hello-world-ansible.yml
    playbook:
       - path: hello-world-ansible.yml

An example with various git file sources (see this entire example at github):

version: 1.0-basic
rspec: docker.rspec

upload:
    # Fetch and upload a single file from a github repo
    - git: 
         url: git@github.com:wvdemeer/espec-test.git
         branch: master
         file: hello3.txt
    # Fetch and upload a file from a subdir of a github repo
    - github: 
         url: git@github.com:wvdemeer/espec-test.git
         dir: hello4
         branch: master
         file: hello4.txt
    # Fetch and upload an entire subdir from a github repo
    - git: 
         url: git@github.com:wvdemeer/espec-test.git
         dir: hello56
         branch: master
    - hello78
    - hello9.txt

ansible:
   host: 
     type: EXISTING
     name: ansible
     upload: hello2.txt
   playbook: hello-playbook.yml

Upload details

Each upload entry supports the following options:

  • a content source (MANDATORY)
  • path (string)
  • permissions (string)
  • nodes (empty, string, or list of string)

path specifies the path on the remote of the file to be uploaded. If no path is specified, the file is uploaded to the upload dir from the dir section. Relative paths are relative to the upload dir from the dir section.

permissions are the required permissions for the uploaded file. If not specified, a default value of u=r is used. The supported format is the same as for dir.

nodes is as for dir and specifies the nodes to which the file needs to be uploaded.

Note that for uploading a file, the permissions of the target dir must allow write access to the login user.

Execute details

Each execute entry contains the following options:

  • a content source
  • path (string)
  • pwd (string)
  • permissions (string)
  • sudo (string)
  • log (string)
  • nodes (string)

path is as with upload, but by default the script dir is used.

permissions is as with execute, but the default permissions are u=rx.

nodes is as with execute.

pwd follows the same rules as path, but specifies the working dir in which the script needs to be executed.

sudo if not specified, this option is false. If true, the script will be executed using sudo to gain root priviledges. Off course, this requires the nodes to have a correctly configured sudo command.

log follows the same rules as path, but specifies where the log file should be written. By default, the same path and base filename as the script will be used, with the extension replaced by ‘.log’.

Global configuration Options

There are a few global config options that change the default behaviour.

This example shows how to set the config options, and shows the default values (i.e. not providing a config section results in the values in this example being set).

version: 1.0-basic
rspec: experiment.rspec
config:
   default_sudo: false                # use sudo when not specified?
   sudo_user: ~                       # null, so no sudo user passed, which sudo iterprets as root
   default_store_remote_logs: true    # when not overridden, store logs at the node executing a command?
   default_store_local_logs: true     # when not overridden, download logs to local client running the ESpec?
   all_nodes_includes_ansible: false  # when uploading or executing on all nodes, include the ansible node?
execute: test.sh

TODO This is not yet implemented completely.

Ansible Support

The Experiment Specification has support for ansible.

Ansible runs on a control machine, which can be:

  • The local machine (not yet supported by jFed)
  • An existing node in your request RSpec
  • And extra node added to your request RSpec

When no preference is specified, a extra node will be added to the request RSpec.

As with dir, upload and execute, a lot of defaults are implied, and if they are used, the syntax can be greatly simplified.

An minimal example experiment-specification file is:

version: 1.0-basic
rspec: experiment.rspec
ansible: hello-world-ansible.yml

In this example:

  • An extra ansible node is added to the RSpec
  • Ansible is installed on that node
  • The ansible inventory file (and related files) are generated and put on the node
  • The specified (bundled) playbook is copied to the node
  • ansible-playbook is called to execute the playbook

Off course, one can specify multiple playbooks to execute, and ansible options. The following example is equivalent, but specifies the playbook as a single item in a list.

version: 1.0-basic
rspec: experiment.rspec
ansible:
    - hello-world-ansible.yml

The 2 previous examples are short for the full syntax:

version: 1.0-basic
rspec: experiment.rspec
ansible:
    playbook: hello-world-ansible.yml

Or as a list:

version: 1.0-basic
rspec: experiment.rspec
ansible:
    playbook: 
       - hello-world-ansible.yml

When all implied defaults are explicitly specified, the same example becomes:

version: 1.0-basic
rspec: experiment.rspec
dir:
   - path: ~/ansible/
     content: ansible
ansible:
    host:
       type: ADD
       name: ansible
       galaxy-command: ansible-galaxy
       playbook-command: ansible-playbook
       upload:
          - generated: keypair
       execute: 
          - download: https://raw.githubusercontent.com/imec-ilabt/ansible-init-script/master/install-ansible.sh
    galaxy: ~
    playbook: 
       - bundled: hello-world-ansible.yml
         debug: 0
         extra-vars: ~
         extra-vars-from: ~
         log: playbook-exec-hello-world-ansible.log

This form explicitly shows most of what adding ansible does. The only thing that is not visible here, is that the ansible inventory file and related files are copied to the ansible dir (~/ansible/ by default).

Not that you can add any upload and execute steps to the ansible host. Rules to take into account when doing so are:

  • The keypair will always be uploaded, even if you do not specify it.
  • The default automatic ansible install script will NOT be used if you specify any custom execute step.
  • When no path is specified, the files are copied to the ansible dir (~/ansible/ by default).

The typical use of these upload and execute steps are to copy files needed by the ansible playbooks, and to install ansible on the node.

The ansible host type, can have 3 values:

  • ADD: A node with the name specified in the name field will be added to the RSpec and used as ansible control machine.
  • EXISTING: A node already present in the RSpec, whose name matches the name field, will be used as ansible control machine.
  • LOCAL: The local host will be used as ansible control machine. The name field should not be specified, and the execute and upload steps are not allowed. (not yet supported by jFed)

Some things will not run on the ansible control machine:

  • The ansible control machine is the only node that will perform the upload and execute steps specified inside ansible. The nodes list of these steps is ignored and should not be specified.
  • The global upload and execute steps that should run on all nodes (no nodes field specified) will NOT run on the ansible control machine, unless the ansible node is of type EXISTING. Only if the ansible control machine is explicitly mentioned in a node list, it will be included.

The playbook options allows specifying one or more playbooks. Each of these requires a source, but also allows additional option: debug, extra-vars and extra-vars-from:

  • The debug option for ansible playbooks is used to determine the number of “v” arguments added to the ansible command. Ansible adds extra debug info for each “v” argument. You can add 0, 1, 2 or 3 “v” arguments.
  • The extra-vars options takes either a string, or a submap. In the case a string is given, this is passed directly to the ansible-playbook --extra-vars option. A map is converted to json and sotred to a file, that is passed using the same option, but with the “@file” syntax: --extra-vars '@filename'
  • The extra-vars-from option also uses the “@file” syntax but allows any source.

An example for the extra-vars options:

playbook:
       - bundled: hello-world1-ansible.yml
         extra-vars: myvar=hello myothervar=hello2
       - bundled: hello-world2-ansible.yml
         extra-vars:
            myvar: hello
            myothervar: hello2
       - bundled: hello-world3-ansible.yml
         extra-vars-from: extraVars.json
       - bundled: hello-world4-ansible.yml
         extra-vars-from: 
            download: http://example.com/my_extra_vars.yml

The galaxy option is similar to the playbook option, but will run before the playbook option, and will call ansible-galaxy to install the requested ansible requirement files. This is used to install ansible modules that add extra features to ansible. In the example above, the galaxy option is null, and thus empty. It expects a list of files like execute and playbook.

The group options allows specifying a list of groups for the ansible inventory file (sometimes called “ansible hosts file”), and the client IDs of the nodes that belong to them. If a group is defined in both the RSpec and the ESpec, the ESpec overrides the group. Otherwise, both sets of groups are added.

A more complex ansible example, with many defauls overridden:

version: 1.0-basic
rspec: my-experiment.rspec
dir:
   - path: /work/ansible/
     content: ansible
ansible:
    host:
       type: EXISTING
       name: control
       galaxy-command: /usr/local/bin/ansible-galaxy
       playbook-command: /usr/local/bin/ansible-playbook
       execute: 
          - my-custom-ansible-install.sh 
       upload:
          - content_that_ansible_needs.txt
    galaxy: 
       - download: http://example.com/ansible-requirements.yml
       - my-ansible-requirements.yml
    playbook: 
       - bundled: setup-software.yml
         debug: 2
       - run-1st-experiment.yml
       - run-2nd-experiment.yml
    group:
       servers:
         - server1
         - server2
       clients:
         - client1
         - client2

ESpec Output

When running an ESpec, it can be handy to output some experiment metadata to file.

You can specify zero, one or more outputs. Each output has a type, which determines what the output is. Each output has a destination which determines to which local file it is written. destination also supports the special values LOG_DEBUG and LOG_INFO to sned the output to the jFed logs instead.

An example with all types of outputs:

version: 1.0-basic
rspec: nodes.rspec
upload: 
   - exp-files-set1.tar.gz
   - exp-files-set2.zip
output:
  - type: ANSIBLE_INVENTORY
    destination: 'ansible-inventory'
  - type: ANSIBLE_PRIVATE_KEY
    destination: 'ansible-privkey.pem'
  - type: ANSIBLE_PUBLIC_KEY
    destination: 'ansible-pubkey.pem'
  - type: ANSIBLE_ZIP
    destination: 'ansible_files.zip'
  - type: ANSIBLE_FILES
    destination: 'ansible_files/'
  - type: REQUEST_RSPEC
    destination: 'request_rspec.xml'
  - type: MANIFEST_RSPEC
    destination: 'manifest_rspec.xml'
  - type: SSH_LOGIN_LIST
    destination: 'ssh_login_list.txt'
  - type: SSH_HOST_LIST
    destination: 'ssh_host_list.txt'
  - type: SSH_INFO_CSV
    destination: 'ssh_info.csv'
  - type: SSH_INFO_JSON
    destination: 'ssh_info.json'
  - type: CLIENT_ID_LIST
    destination: '/tmp/client_ids.txt'
  - type: SSH_HOST_LIST
    destination: LOG_DEBUG
  - type: CLIENT_ID_LIST
    destination: LOG_INFO

Using an ESpec in jFed

The jFed Experiment GUI currently supports ESpecs. Look for the “Open ESpec” button. You can select an archive, directory, URL or github repo. Next, you’ll be asked to choose a project and slice.

The jFed Experiment CLI 2 has support for running ESpecs. Check the documentation for more info.

The jFed Automated tester, and thus fedmon, supports ESpec in both the GuiLogicTest and ESpecTest.