Command Line Interface¶

After installation, midtools can be run from the command line. This is a summary of the parser arguments:

Analyze MID runs.

usage: midtools [-h] [-r RUN] [-dr DARK_RUN_NUMBER DARK_RUN_NUMBER]
                [--last-train LAST_TRAIN] [--first-train FIRST_TRAIN]
                [-ppt PULSES_PER_TRAIN] [-ts TRAIN_STEP] [-ps PULSE_STEP]
                [--is-dark [IS_DARK]] [--is-flatfield [IS_FLATFIELD]]
                [-ffr FLATFIELD_RUN FLATFIELD_RUN] [--out-dir [OUT_DIR]]
                [--chunk [CHUNK]] [--job-dir JOB_DIR] [--slurm [SLURM]]
                [--localcluster [LOCALCLUSTER]]
                [--file-identifier [FILE_IDENTIFIER]]
                [--first-cell FIRST_CELL] [--datdir DATDIR]
                setupfile analysis

Positional Arguments¶

`setupfile`	the YAML file to configure midtools
`analysis`	which analysis to perform. List of 0s and 1s: 1000 saves average data along specific axis, 0100 SAXS routines, 0010 XPCS routines, 0001 statistics (histograms pulse resolved.

Named Arguments¶

`-r, --run`	Run number.
`-dr, --dark-run-number`
	Dark run number.
`--last-train`	last train to analyze. Default: 1000000
`--first-train`	first train to analyze. Default: 0
`-ppt, --pulses-per-train`
	number of pulses per train Default: 500
`-ts, --train-step`
	spacing of trains Default: 1
`-ps, --pulse-step`
	spacing of pulses Default: 1
`--is-dark`	whether the run is a dark run Default: False
`--is-flatfield`	whether the run is a flatfield run Default: False
`-ffr, --flatfield-run`
	Flatfield run number.
`--out-dir`	Output directory Default: “./”
`--chunk`	Split the number of trains in chunks of this size (default do not chunk)
`--job-dir`	Directory for the slurm output and error files Default: “/gpfs/exfel/data/scratch/reiserm/mid-proteins/jobs/”
`--slurm`	Run midtools on dedicated node with slurm job (default False) Default: False
`--localcluster`	Use dasks LocalCluster to run midtools locally (default False) Default: False
`--file-identifier`
	Identifier at file ending. Default None.
`--first-cell`	Cell ID of the first AGIPD memory cell with X-rays. Default: 2
`--datdir`	Path to the data. This argument is only used if the data directory is not provided in the setupfile.

For example, the SAXS routine can be run on each pulse of a run by:

>>> midtools /path/to/setupfile/setup.yml 10

The setup.yml file is a necessary argument. It contains information on the current experimental setup, the data directory, etc. and defines other parameters important for the analysis routines like the q-bins for the XPCS analysis. The next argument is a serious of ones or zeros that determines which analysis should be performed. See The Dataset Class for more details on the arguments and their purpose.

Warning

The run number has to be specified in the setupfile by giving the full path to the data including the run folder /path/r0123. The run directory can be ommited if the run argument is passed on the command line.

If the run directory is not included in the setupfile, it has to be specified. For example, this would analyze the first 1000 trains of run 123 and run SAXS and XPCS routines:

>>> midtools /path/to/setupfile/setup.yml 11 --last 1000 --run 123

Additinally, the number of X-ray pulses per train can be specified:

>>> midtools /path/to/setupfile/setup.yml 11 --last 1000 --run 123 --ppt 100

This would assume that 100 pulses per train (ppt) were deliverd by the machine.

Note

If you are insecure which arguments you should provide, pass all you know.

Working With Raw Data¶

Midtools comes with simple data calibration routines for HG (high-gain) AGIPD data including:

dark subtraction,
creating of a pixel mask based on dark and flatfield data,
baseline shift based on Ta-stripes,
flexible (spatial) commonmode correction (default: asic level 64x64 pixels),
dropletizing by thresholding (converting ADUs to photons).

To work with raw data, a dark run has to be processed by passing the –is-dark-flag:

>>> midtools setup.yml 11 --last 500 --run 001 --ppt 200 --is-dark

The output file would be r0001-dark_000.h5 (the counter might not be 000).

To use a previously created dark file, pass the –dark-run argument. The argument requires both the run number and the file index. So if you want to use the dark file r0001-dark_000.h5, you would run midtools like:

>>> midtools setup.yml 11 --last 500 --run 011 --ppt 20 --dark-run 1 0

Warning

Make sure to use all pulses when processing the dark run.

Latest Changes and Remarks¶

midtools can read the pulse pattern from the machine control sources which in principle allows to omit the –ppt or –pulses-per-train argument. However, this source is not always available in the DAQ. Passing the argument manually will also overwride the settings read from the control source.