scingestor package

Submodules

scingestor.beamtimeWatcher module

class scingestor.beamtimeWatcher.BeamtimeWatcher(options)

Bases: object

Beamtime Watcher

constructor

Parameters:

options (argparse.Namespace) – parser options

running

(bool) running loop flag

start()

start beamtime watcher

stop()

stop beamtime watcher

scingestor.beamtimeWatcher.main(interrupt=0)

the main program function

Parameters:

interrupt (int) – test interrupt flag: 1:keyboard, 2:signal

scingestor.configuration module

scingestor.configuration.load_config(configfile)

load config file

Parameters:

configfile (str) – configuration file name

scingestor.datasetIngest module

class scingestor.datasetIngest.DatasetIngest(options)

Bases: object

Dataset Ingest command

constructor

Parameters:

options (argparse.Namespace) – parser options

start()

start ingestion

scingestor.datasetIngest.main()

the main program function

scingestor.datasetIngestor module

class scingestor.datasetIngestor.DatasetIngestor(configuration, path, dsfile, idsfile, meta, beamtimefile)

Bases: object

Dataset Ingestor

constructor

Parameters:

• configuration (dict <str, any>) – dictionary with the ingestor configuration

• path (str) – scan dir path

• dsfile (str) – file with a dataset list

• dsfile – file with a ingester dataset list

• meta (dict <str, any>) – beamtime configuration

• beamtimefile (str) – beamtime filename

• pidprefix (str) – pidprefix

• ingestorcred (str) – ingestor credential

• scicat_url (str) – scicat_url

append_proposal_groups()

appends owner and access groups to beamtime

Parameters:

• meta (dict <str, any>) – beamtime configuration

• path (str) – base file path

Returns:

updated beamtime configuration

Return type:

dict <str, any>

check_list(reingest=False)

update waiting and ingested datasets

clear_tmpfile()

clear waitings datasets

clear_waiting_datasets()

clear waitings datasets

get_token()

provides ingestor token

Returns:

ingestor token

Return type:

str

ingest(scan, token)

ingest scan

Parameters:

• scan (str) – scan name

• token (str) – access token

ingested_datasets()

provides ingested datasets

Returns:

ingested datasets list

Return type:

list <str>

reingest(scan, token, notmp=False)

re-ingest scan

Parameters:

• scan (str) – scan name

• token (book) – access token

• token – no tmp file flag

start_measurement(measurement)

start measurement

Parameters:

measurement (str) – measurement name

stop_measurement()

stop measurement

update_from_tmpfile()

clear waitings datasets

waiting_datasets()

provides waitings datasets

Returns:

waitings datasets list

Return type:

list <str>

class scingestor.datasetIngestor.UpdateStrategy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

Update strategy

CREATE = 2

(scingestor.datasetIngestor.UpdateStrategy) recreate datasets

MIXED = 3

(scingestor.datasetIngestor.UpdateStrategy) patch datasets only if scientificMetadata changed otherwise recreate datasets

NO = 0

(scingestor.datasetIngestor.UpdateStrategy) leave datasets unchanged

PATCH = 1

(scingestor.datasetIngestor.UpdateStrategy) patch datasets

scingestor.datasetWatcher module

class scingestor.datasetWatcher.DatasetWatcher(configuration, path, dsfile, idsfile, meta, beamtimefile)

Bases: Thread

Dataset Watcher

constructor

Parameters:

• configuration (dict <str, any>) – dictionary with the ingestor configuration

• path (str) – scan dir path

• dsfile (str) – file with a dataset list

• dsfile – file with a ingester dataset list

• meta (dict <str, any>) – beamtime configuration

• beamtimefile (str) – beamtime filename

run()

scandir watcher thread

running

(bool) running loop flag

stop()

stop the watcher

scingestor.logger module

class scingestor.logger.AccSecFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Bases: Formatter

micro-second formatter

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

Changed in version 3.2: Added the style parameter.

converter()

timestamp[, tz] -> tz’s local time from POSIX timestamp.

formatTime(record, datefmt=None)

Return the creation time of the specified LogRecord as formatted text.

This method should be called from format() by a formatter which wants to make use of a formatted time. This method can be overridden in formatters to provide for any specific requirement, but the basic behaviour is as follows: if datefmt (a string) is specified, it is used with time.strftime() to format the creation time of the record. Otherwise, an ISO8601-like (or RFC 3339-like) format is used. The resulting string is returned. This function uses a user-configurable function to convert the creation time to a tuple. By default, time.localtime() is used; to change this for a particular formatter instance, set the ‘converter’ attribute to a function with the same signature as time.localtime() or time.gmtime(). To change it for all formatters, for example if you want all logging times to be shown in GMT, set the ‘converter’ attribute in the Formatter class.

scingestor.logger.get_logger()

provides logger object

Return type:

logging.logger

Returns:

logger object

scingestor.logger.init_logger(name='scingestor.logger', level='debug', timestamps=False, logfile=None)

init logger

Parameters:

• name (str) – logger name

• level (str) – logging level

• timestamps (bool) – timestamps flag

• logfile (str) – logger file name

scingestor.modelIngest module

class scingestor.modelIngest.ModelIngest(options)

Bases: object

Dataset Ingest command

constructor

Parameters:

options (argparse.Namespace) – parser options

get_token()

provides ingestor token

Returns:

ingestor token

Return type:

str

start()

start ingestion

scingestor.modelIngest.main()

the main program function

scingestor.pathConverter module

class scingestor.pathConverter.PathConverter(corepath, blpath, usecorepath=False)

Bases: object

Path Converter

constructor

Parameters:

• corepath (str) – core path

• blpath (str) – beamline path

• usecorepath (bool) – enabled flag

from_core(path)

converts core path to notify path

Parameters:

path (str) – core path

Returns:

notify path

Return type:

str

to_core(path)

converts notify path to core path

Parameters:

path (str) – notify path

Returns:

core path

Return type:

str

scingestor.safeINotifier module

class scingestor.safeINotifier.EventData(name, masks)

Bases: object

event data

constructor

Parameters:

• name (str) – name

• masks – mask description

masks

(str) mask

name

(str) name

class scingestor.safeINotifier.SafeINotifier(*args, **kwargs)

Bases: Thread

singleton wrapper for inotifyx

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is a list or tuple of arguments for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.

add_watch(path, masks)

add watch to notifier

Parameters:

• path (str) – watch path

• mask (int) – watch mask

Returns:

queue providing events and its id

Return type:

[queue.Queue, int]

daemon = True

(bool) make notifier to be a daemon

init()

constructor

rm_watch(qid)

remove watch from notifier

Parameters:

qid (int) – queue id

run()

scandir watcher thread

stop()

stop the watcher

scingestor.scanDirWatcher module

class scingestor.scanDirWatcher.ScanDirWatcher(configuration, path, meta, beamtimefile, depth)

Bases: Thread

ScanDir Watcher

constructor

Parameters:

• configuration (dict <str, any>) – dictionary with the ingestor configuration

• path (str) – scan dir path

• meta (dict <str, any>) – beamtime configuration

• beamtimefile (str) – beamtime file

• depth (int) – scandir depth level

run()

scandir watcher thread

running

(bool) running loop flag

stop()

stop the watcher

Module contents