definitions package

definitions package#

Package that defines variables used within this repository. definitions.dmoore is not imported in order not to have to import Moore.

definitions.dallen module#

Variables used for the Allen algorithms.

definitions.dallen.all_mdf2csv_algos = ['hits_velo', 'hits_ut', 'hits_scifi', 'mc_particles', 'mc_vertices', 'mc_hit_links']#: List of the possible Allen algorithms

definitions.dcomputing module#

Some variables used for Ganga and HTCondor integration.

definitions.dcomputing.backends = ['local', 'condor', 'ganga-local', 'ganga-dirac']#: List of all the backends

definitions.dcomputing.environment_variables_to_copy = {'default': ['XDIGI2CSV_REPO', 'X509_USER_PROXY', 'HOME'], 'ganga-grid': ['XDIGI2CSV_REPO'], 'ganga-local': ['XDIGI2CSV_REPO']}#: Associates a “backend” with the environment variables to pass to the backend

definitions.dcomputing.ganga_backends = ['ganga-local', 'ganga-dirac']#: List of the ganga backends

definitions.dconfig module#

Define the configuration parameters.

definitions.dconfig.config_description = {'allen_config': {'events_per_slice': {'help': 'Number of events per slice.', 'type': <class 'int'>}, 'memory': {'help': 'Memory to reserve on the device per thread / stream (megabytes)', 'type': <class 'int'>}, 'number_of_events': {'help': 'Number of events to process.', 'type': <class 'int'>}, 'number_of_slices': {'help': 'Number of input slices to allocate', 'type': <class 'int'>}, 'repetitions': {'help': 'number of threads / streams.', 'type': <class 'int'>}, 'treads': {'help': 'Number of threads.', 'type': <class 'int'>}}, 'allen_input': {'geo_dirname': {'help': 'Geometry directory name inside `indir`'}, 'geodir': {'help': 'Directory where the binary geometry files are for Allen'}, 'indir': {'help': 'Input directory where the MDF files and possibly the geometry directory are'}, 'mdf_filename': {'help': 'If `indir` is used, MDF file name (can be `*.mdf`)'}, 'paths': {'help': 'MDF input files for Allen', 'nargs': '+'}}, 'build': {'allen_standalone': {'help': 'Path to the build directory of Allen standalone'}, 'moore': {'help': 'Path to the Moore repository within the stack'}, 'platform': {'help': 'Platform of the build to use. This is used only in a Moore program'}}, 'computing': {'backend': {'choices': ['local', 'condor'], 'help': 'Where to run the job. Use `local` for tests or small jobs and `condor` for many jobs.'}, 'max_runtime': {'help': 'Maximal time in sec within which the job is expected to run.', 'type': <class 'int'>}, 'max_transfer_output_mb': {'help': '`MAX_TRANSFER_OUTPUT_MB` HTCondor parameter.', 'type': <class 'int'>}, 'nb_files_per_job': {'help': 'Number of input files per job.'}, 'program': {'choices': ['xdigi2csv', 'xdigi2mdf', 'xdigi2root', 'test', 'mdf2csv', 'velo_validation_no_gec', 'persistence_velo_tracks', 'persistence_seeding_tracks', 'persistence_seeding_xz_tracks', 'seeding_validation_no_gec'], 'help': 'Which program to run.'}, 'wildcard_data_logdir': {'help': 'Where to save the log if output is in EOS, that is, `auto_output_mode` is set to `eos`. This argument is only relevant if HTCondor is used. As the log cannot be written on EOS, another directory must be chosen.'}}, 'global': {'datadir': {'help': 'Path where data is stored in the EOS'}}, 'mdf2csv': {'algos': {'choices': ['hits_velo', 'hits_ut', 'hits_scifi', 'mc_particles', 'mc_vertices', 'mc_hit_links'], 'help': 'List of the Allen algorithms to run', 'nargs': '+'}, 'erase': {'help': 'Whether to erase an existing CSV file', 'type': <class 'bool'>}}, 'moore_input': {'banned_storage_elements': {'help': 'List of storage elements to ban', 'nargs': '+'}, 'bookkeeping_path': {'help': 'Bookkeeping path. The actual PFNs will be determined automatically'}, 'conddb_tag': {'help': 'Version of the conditions database to be used, which specifies the calibration and alignment constants of the LHCb detector.'}, 'data_type': {'help': 'Data-taking year the sample is supposed to represent.'}, 'dddb_tag': {'help': 'Version of the Detector Description DataBase, which specifies the geometry and material of the LHCb detector'}, 'evt_max': {'help': 'Maximal number of events.', 'type': <class 'int'>}, 'nb_files': {'help': 'Number of files to process. -1 for all the files.', 'type': <class 'int'>}, 'paths': {'help': 'Path to the files, directly', 'nargs': '+'}, 'python_input': {'help': 'Path to the python file that defines the input. It is handled by Moore'}, 'simulation': {'help': 'Whether the sample is a simulation. ``Upgrade`` for the run 3.'}, 'start_index': {'help': 'Index of the file from which to start', 'type': <class 'int'>}}, 'output': {'auto_output_mode': {'help': '`outdir` if `outdir` is used, `eos` if `wildcard_data_outdir` is used'}, 'dataname': {'help': 'Input data name'}, 'logpath': {'help': 'If set, the stdout and stderr will be written to this file.'}, 'outdir': {'help': 'Path to the output directory'}, 'suboutdir': {'help': 'Sub-directory where to save the files'}, 'version': {'help': 'Software version'}, 'wildcard_data_outdir': {'help': 'Output paths, with placeholders `version`, `dataname` and `datatype`. `datatype` would be `csv` for CSV outputs'}}, 'snakemake': {'datadir_in': {'help': 'Input data directory to use as input for the scripts (need fast access)'}, 'datadir_out': {'help': 'Input data directory to use as output for the scripts'}}, 'test': {}, 'xdigi2csv': {'_intermediate_outdir_in_node': {'help': 'If set to `True`, a `None` value of `intermediate_outdir` with `keep_original` set to `False` is interpreted as writting on the local space of condor. This variable is set by the `run/run.py` script but you can force it to a value (`true` or `false`).', 'type': <class 'bool'>}, 'all_mc_particles': {'help': 'whether to dump all MC particles, even those which do not have hits in the selected detectors', 'type': <class 'bool'>}, 'compression': {'help': 'Compression algorithm to use. Availability depends on the chosen format.'}, 'detectors': {'choices': ['velo', 'ut', 'scifi'], 'help': 'List of detectors for which the hits and MC particles are dumped', 'nargs': '+'}, 'dump_event_info': {'help': 'whether to dump `event_info.csv`', 'type': <class 'bool'>}, 'dump_log': {'help': 'Decide on saving a YAML file in the output folder that details the input and records the success of the program.', 'type': <class 'bool'>}, 'dump_mc_hits': {'help': 'whether to dump the MC hits, i.e., `mchits_{detector}.csv` for the selected detectors', 'type': <class 'bool'>}, 'erase': {'help': 'Whether to erase an existing CSV file', 'type': <class 'bool'>}, 'extended': {'help': 'Whether to dump more information for the `hits_{detector}.csv` and `mc_particles.csv`', 'type': <class 'bool'>}, 'format': {'choices': ['csv', 'feather', 'parquet', 'root', 'rroot'], 'help': 'Output format.'}, 'intermediate_outdir': {'help': 'Intermediate output dir²ectory where to store the CSV file before the conversion. You can set this to `.` No value means that the intermediate output directory is the same as the output directory.'}, 'keep_original': {'help': 'In the case were the file is compressed or his format is changed, whether to keep the original.', 'type': <class 'bool'>}, 'retina_clusters': {'help': 'Whether retina clusters are in the (X)DIGI file', 'type': <class 'bool'>}}, 'xdigi2mdf': {'geo_dirname': {'help': 'Directory name of directory where the binary geometry files are dumped'}, 'mdf_filename': {'help': 'File name of the outputted MDF file'}, 'retina_clusters': {'help': 'Whether retina clusters are in the (X)DIGI file', 'type': <class 'bool'>}}, 'xdigi2root': {'retina_clusters': {'help': 'Whether retina clusters are in the (X)DIGI file', 'type': <class 'bool'>}}}#: Associates a section in the configuration with another dictionnary that associates an option with a dictionnary that will be passed to argparse.ArgumentParser.add_argument(). Use nargs="+" for lists and type=bool for a boolean variable

definitions.dconfig.config_sections = {'allen_config': 'Allen configuration', 'allen_input': 'Input of an Allen program', 'build': 'Path to the build directories of Moore and Allen standalone', 'computing': 'Configuration for running jobs', 'global': 'Global variables', 'mdf2csv': 'Conversion from MDF to CSV in Allen (experimental)', 'moore_input': 'Input of a Moore program. It is either provided by a python file (`python_input`), a list of PFNs (`paths`), or a unique bookkeeping path (`bookkeeping_path`) together with `start_index`, `nb_files`, `dddb_tag`, `conddb_tag`, `simulation`, `data_type` and `evt_max`', 'output': 'Output directory. Either you can set ``outdir``, or use ``auto_output_mode`` (in this case, ``outdir`` is overwritten)', 'persistence_seeding_tracks': 'Allen sequence to dump the seeding tracks into a CSV file.', 'persistence_seeding_xz_tracks': 'Allen sequence to dump the seeding XZ tracks into a CSV file.', 'persistence_velo_tracks': 'Allen sequence to dump the velo tracks into a CSV file.', 'seeding_validation_no_gec': 'Allen seeding validation sequence, without the General Event Cut (GEC)', 'snakemake': 'Configuration of the scripts run using snakemakes', 'test': 'Test program for Moore', 'velo_validation_no_gec': 'Allen velo validation sequence, without the General Event Cut (GEC)', 'xdigi2csv': 'Conversion from (X)DIGI to CSV', 'xdigi2mdf': 'Conversion from (X)DIGI to MDF', 'xdigi2root': 'Conversion from (X)DIGI to ROOT'}#: Associates a section in the configuration with a description

definitions.dconfig.input_config_params = ['dddb_tag', 'conddb_tag', 'simulation', 'data_type', 'evt_max']#: List of information needed to create the input file, that need to be stored in the configuration file

definitions.dconfig.moore_programs = ['xdigi2csv', 'xdigi2mdf', 'xdigi2root', 'test']#: List of Moore programs, run with ./run/moore/run.py

definitions.dconfig.path_list_location = [{'moore_input': ['paths']}, {'allen_input': ['indir', 'paths', 'geodir']}, {'output': ['outdir', 'logpath']}, {'build': ['moore', 'allen_standalone']}, 'include', {'snakemake': ['datadir_in', 'datadir_out']}]#: A list of elements that indicate the value that correspond to a path or a list of paths, that could be relative and need to be resolved.

definitions.dconfig.program_to_datatype = {'mdf2csv': 'csv', 'xdigi2csv': 'csv', 'xdigi2mdf': 'mdf', 'xdigi2root': 'root'}#: Associates a program with the value of the variable datatype which is used in the path of the output directory if saved in the EOS

definitions.dconfig.program_to_input_section = {'mdf2csv': 'allen_input', 'persistence_seeding_tracks': 'allen_input', 'persistence_seeding_xz_tracks': 'allen_input', 'persistence_velo_tracks': 'allen_input', 'seeding_validation_no_gec': 'allen_input', 'test': 'moore_input', 'velo_validation_no_gec': 'allen_input', 'xdigi2csv': 'moore_input', 'xdigi2mdf': 'moore_input', 'xdigi2root': 'moore_input'}#: Associates a program of this repo with the relevant input section in the configuration

definitions.dconfig.programs = ['xdigi2csv', 'xdigi2mdf', 'xdigi2root', 'test', 'mdf2csv', 'velo_validation_no_gec', 'persistence_velo_tracks', 'persistence_seeding_tracks', 'persistence_seeding_xz_tracks', 'seeding_validation_no_gec']#: List of programs in this repository

definitions.dconversion module#

Define functions and tools used for conversion.

definitions.dconversion.compression_exts = {'gzip': 'gz', 'zstd': 'zst'}#: Associates a compression method with an extension

definitions.dconversion.format_compressions = {'csv': ['gzip', 'lz4', 'zstd', 'bz2', 'brotli'], 'feather': ['uncompressed', 'lz4', 'zstd'], 'parquet': ['none', 'snappy', 'gzip', 'zstd', 'lz4'], 'root': ['zlib', 'lzma', 'lz4', 'zstd'], 'rroot': ['zlib', 'lzma', 'lz4', 'zstd']}#: Associates a format with the available compressions

definitions.dconversion.format_no_compressions = {'feather': 'uncompressed', 'parquet': 'none'}#: Associates a format with a compression that actually means that there is no compression

definitions.dconversion.formats = ['csv', 'feather', 'parquet', 'root', 'rroot']#: Possible formats

definitions.dconversion.get_extension(format, compression=None)[source]#

Get compression from format and compression.

Parameters:

format (str) – csv, feather or parquet
compression (Optional[str]) – compression algorithm to use. See format_compressions for the possible compression given a format.

Return type:

str

Returns:

Extension of the file given its format and compression

definitions.dconversion.get_io_function(action, format)[source]#

Get the function to read or write in a given format

Parameters:

action (Literal['r', 'w', 'read', 'write']) –
What to return
- r or read for the reading function
- w or write for the writting function
format (str) – csv, feather or parquet

Return type:

Tuple[Callable, dict]

Returns:

The function to read or write the given format using pyarrow. For the writting function, the compression can be provided using the keyword argument compression of the function that is returned. The table to write is the first argument and the path is the second. For the reading function, the compression is figured out from the extension of the file

definitions.dconversion.root_compression_levels = {'lz4': 4, 'lzma': 8, 'zlib': 1, 'zstd': 5}#: Compression levels of a ROOT file according to the chosen compression algorithm, as recommended in https://root.cern/doc/master/Compression_8h_source.html

definitions.dgen module#

Module that defines general variables.

definitions.dgen.all_algonames = ['mc_particles', 'event_info', 'hits_velo', 'hits_ut', 'hits_scifi', 'mchits_velo', 'mchits_ut', 'mchits_scifi']#: All algorithm names (and CSV output names)

definitions.dgen.all_detectors = ['velo', 'ut', 'scifi']#: Tracking detectors

definitions.dmoore module#

Define the algorithms to use in Moore.

definitions.dmoore.algoname_to_algo = {'event_info': RecoConf.persistence_csv.csv_persistence_event_info, 'hits_scifi': RecoConf.persistence_csv.csv_persistence_ft_hits, 'hits_ut': RecoConf.persistence_csv.csv_persistence_ut_hits, 'hits_velo': RecoConf.persistence_csv.csv_persistence_vp_hits, 'mc_particles': RecoConf.persistence_csv.csv_persistence_mc_particles, 'mchits_scifi': RecoConf.persistence_csv.csv_persistence_ft_mchits, 'mchits_ut': RecoConf.persistence_csv.csv_persistence_ut_mchits, 'mchits_velo': RecoConf.persistence_csv.csv_persistence_vp_mchits}#: Associates an algorithm name with the actual configurable function in Moore

definitions.dpaths module#

Module that defines the paths to the scripts.