Df Memory

class dffml.df.memory.BaseMemoryDataFlowObject(config: Optional[Type[dffml.base.BaseConfig]])

class dffml.df.memory.MemoryDataFlowObjectContextConfig(uid: str)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryDefinitionSetContext(config: dffml.base.BaseConfig, parent: dffml.df.base.BaseInputNetworkContext, ctx: dffml.df.base.BaseInputSetContext)

async inputs(definition: dffml.df.types.Definition) → AsyncIterator[dffml.df.types.Input]

Asynchronous iterator of all inputs within a context, which are of a definition.

class dffml.df.memory.MemoryInputNetwork(config: Optional[Type[dffml.base.BaseConfig]])

Inputs backed by a set

CONFIG

alias of dffml.df.memory.MemoryInputNetworkConfig

CONTEXT

alias of dffml.df.memory.MemoryInputNetworkContext

class dffml.df.memory.MemoryInputNetworkConfig

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryInputNetworkContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryInputNetwork)

async add(input_set: dffml.df.base.BaseInputSet)

Adds new input set to the network

async added(watch_ctx: dffml.df.base.BaseInputSetContext) → Tuple[bool, dffml.df.base.BaseInputSet]

Returns when a new input set has entered the network within a context

async cadd(ctx, *args: dffml.df.types.Input)

Shorthand for creating a MemoryInputSet with an existing context.

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemoryOrchestrator() as orchestrator:
...         async with orchestrator(DataFlow.auto()) as octx:
...             await octx.ictx.sadd(StringInputSetContext("Hi"))
>>>
>>> asyncio.run(main())

async ctx() → Tuple[bool, dffml.df.base.BaseInputSetContext]

Returns when a new input set context has entered the network

async definition(ctx: dffml.df.base.BaseInputSetContext, definition: str) → dffml.df.types.Definition

Search for the definition within a context given its name as a string. Return the definition. Otherwise raise a DefinitionNotInContext error. If the context is not present, raise a ContextNotPresent error.

definitions(ctx: dffml.df.base.BaseInputSetContext) → dffml.df.base.BaseDefinitionSetContext

Return a DefinitionSet context that can be used to access the inputs within the given context, by definition.

async gather_inputs(rctx: dffml.df.base.BaseRedundancyCheckerContext, operation: dffml.df.types.Operation, dataflow: dffml.df.types.DataFlow, ctx: Optional[dffml.df.base.BaseInputSetContext] = None) → AsyncIterator[dffml.df.base.BaseParameterSet]

Generate all possible permutations of applicable inputs for an operation that, according to the redundancy checker, haven’t been run yet.

async receive_from_parent_flow(inputs: List[dffml.df.types.Input])

Takes input from parent dataflow and adds it to every active context

async sadd(context_handle_string, *args: dffml.df.types.Input)

Shorthand for creating a MemoryInputSet with a StringInputSetContext.

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemoryOrchestrator() as orchestrator:
...         async with orchestrator(DataFlow.auto()) as octx:
...             await octx.ictx.sadd("Hi")
>>>
>>> asyncio.run(main())

async uadd(*args: dffml.df.types.Input)

Shorthand for creating a MemoryInputSet with a StringInputSetContext containing a random value for the string.

class dffml.df.memory.MemoryInputNetworkContextEntry(ctx, definitions, by_origin)

property by_origin

Alias for field number 2

property ctx

Alias for field number 0

property definitions

Alias for field number 1

class dffml.df.memory.MemoryInputSet(config: dffml.df.memory.MemoryInputSetConfig)

async add(item: dffml.df.types.Input) → None

Add an input to the input set.

async remove_input(item: dffml.df.types.Input)

Removes item from input set

async remove_unvalidated_inputs() → dffml.df.memory.MemoryInputSet

Removes unvalidated inputs from internal list and returns the same.

class dffml.df.memory.MemoryInputSetConfig(ctx: dffml.df.base.BaseInputSetContext, inputs: List[dffml.df.types.Input])

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryKeyValueStore(config: Optional[Type[dffml.base.BaseConfig]])

Key Value store backed by dict

CONFIG

alias of dffml.df.memory.MemoryKeyValueStoreConfig

CONTEXT

alias of dffml.df.memory.MemoryKeyValueStoreContext

class dffml.df.memory.MemoryKeyValueStoreConfig

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryKeyValueStoreContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryKeyValueStore)

async get(key: str) → Optional[bytes]

Get a value from the key value store

async set(key: str, value: bytes)

Get a value in the key value store

class dffml.df.memory.MemoryLockNetwork(config: Optional[Type[dffml.base.BaseConfig]])

CONFIG

alias of dffml.df.memory.MemoryLockNetworkConfig

CONTEXT

alias of dffml.df.memory.MemoryLockNetworkContext

class dffml.df.memory.MemoryLockNetworkConfig

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryLockNetworkContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryLockNetwork)

acquire(parameter_set: dffml.df.base.BaseParameterSet)

Acquire the lock for each input in the input set which must be locked prior to running an operation using the input.

class dffml.df.memory.MemoryOperationImplementationNetwork(config: Optional[Type[dffml.base.BaseConfig]])

CONFIG

alias of dffml.df.memory.MemoryOperationImplementationNetworkConfig

CONTEXT

alias of dffml.df.memory.MemoryOperationImplementationNetworkContext

class dffml.df.memory.MemoryOperationImplementationNetworkConfig(operations: Dict[str, dffml.df.base.OperationImplementation] = <factory>)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryOperationImplementationNetworkContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryOperationImplementationNetwork)

async contains(operation: dffml.df.types.Operation) → bool

Checks if operation in is operations we have loaded in memory

async dispatch(octx: dffml.df.base.BaseOrchestratorContext, operation: dffml.df.types.Operation, parameter_set: dffml.df.base.BaseParameterSet)

Schedule the running of an operation

async ensure_contains(operation: dffml.df.types.Operation)

Raise errors if we don’t have and can’t instantiate an operation.

async instantiable(operation: dffml.df.types.Operation, *, opimp: Optional[dffml.df.base.OperationImplementation] = None) → bool

Looks for class registered with ____ entrypoint using pkg_resources.

async instantiate(operation: dffml.df.types.Operation, config: dffml.base.BaseConfig, *, opimp: Optional[dffml.df.base.OperationImplementation] = None) → bool

Instantiate class registered with ____ entrypoint using pkg_resources. Return true if instantiation was successful.

async operation_completed()

Returns when an operation finishes

async run(ctx: dffml.df.base.BaseInputSetContext, octx: dffml.df.base.BaseOrchestratorContext, operation: dffml.df.types.Operation, inputs: Dict[str, Any]) → Union[bool, Dict[str, Any]]

Run an operation in our network.

async run_dispatch(octx: dffml.df.base.BaseOrchestratorContext, operation: dffml.df.types.Operation, parameter_set: dffml.df.base.BaseParameterSet, set_valid: bool = True)

Run an operation in the background and add its outputs to the input network when complete

async run_no_retry(ctx: dffml.df.base.BaseInputSetContext, octx: dffml.df.base.BaseOrchestratorContext, operation: dffml.df.types.Operation, inputs: Dict[str, Any]) → Union[bool, Dict[str, Any]]

Run an operation in our network without retry if it fails

class dffml.df.memory.MemoryOperationNetwork(config: Optional[Type[dffml.base.BaseConfig]])

Operations backed by a set

CONFIG

alias of dffml.df.memory.MemoryOperationNetworkConfig

CONTEXT

alias of dffml.df.memory.MemoryOperationNetworkContext

class dffml.df.memory.MemoryOperationNetworkConfig(operations: List[dffml.df.types.Operation] = <factory>)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryOperationNetworkContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryOperationNetwork)

async add(operations: List[dffml.df.types.Operation])

Add operations to the network

async operations(dataflow: dffml.df.types.DataFlow, *, input_set: Optional[dffml.df.base.BaseInputSet] = None, stage: dffml.df.types.Stage = Stage.PROCESSING) → AsyncIterator[dffml.df.types.Operation]

Retrieve all operations in the network of a given stage filtering by operations who have inputs with definitions in the input set.

class dffml.df.memory.MemoryOrchestrator(config: Optional[dffml.base.BaseConfig])

CONFIG

alias of dffml.df.memory.MemoryOrchestratorConfig

CONTEXT

alias of dffml.df.memory.MemoryOrchestratorContext

class dffml.df.memory.MemoryOrchestratorConfig(input_network: dffml.df.base.BaseInputNetwork = <factory>, operation_network: dffml.df.base.BaseOperationNetwork = <factory>, lock_network: dffml.df.base.BaseLockNetwork = <factory>, opimp_network: dffml.df.base.BaseOperationImplementationNetwork = <factory>, rchecker: dffml.df.base.BaseRedundancyChecker = <factory>, max_ctxs: int = None)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryOrchestratorContext(config: dffml.df.memory.MemoryOrchestratorContextConfig, parent: dffml.df.base.BaseOrchestrator)

async dispatch_auto_starts(ctx)

Schedule the running of all operations without inputs

async initialize_dataflow(dataflow: dffml.df.types.DataFlow) → None

Initialize a DataFlow by preforming the following steps.

1. Add operations the operation network context

2. Instantiate operation implementations which are not instantiated within the operation implementation network context

3. Seed input network context with given inputs

async operations_parameter_set_pairs(ctx: dffml.df.base.BaseInputSetContext, dataflow: dffml.df.types.DataFlow, *, new_input_set: Optional[dffml.df.base.BaseInputSet] = None, stage: dffml.df.types.Stage = Stage.PROCESSING) → AsyncIterator[Tuple[dffml.df.types.Operation, dffml.df.base.BaseInputSet]]

Use new_input_set to determine which operations in the network might be up for running. Cross check using existing inputs to generate per input set context novel input pairings. Yield novel input pairings along with their operations as they are generated.

async run(*input_sets: Union[List[dffml.df.types.Input], dffml.df.base.BaseInputSet], strict: bool = True, ctx: Optional[dffml.df.base.BaseInputSetContext] = None, halt: Optional[asyncio.locks.Event] = None) → AsyncIterator[Tuple[dffml.df.base.BaseContextHandle, Dict[str, Any]]]

Run a DataFlow.

class dffml.df.memory.MemoryOrchestratorContextConfig(uid: str, dataflow: dffml.df.types.DataFlow, reuse: Dict[str, dffml.df.base.BaseDataFlowObjectContext] = None, max_ctxs: int = None)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryParameterSet(config: dffml.df.memory.MemoryParameterSetConfig)

class dffml.df.memory.MemoryParameterSetConfig(ctx: dffml.df.base.BaseInputSetContext, parameters: List[dffml.df.types.Parameter])

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryRedundancyChecker(config)

Redundancy Checker backed by Memory Key Value Store

CONFIG

alias of dffml.df.memory.MemoryRedundancyCheckerConfig

CONTEXT

alias of dffml.df.memory.MemoryRedundancyCheckerContext

class dffml.df.memory.MemoryRedundancyCheckerConfig(kvstore: dffml.df.base.BaseKeyValueStore = <factory>)

no_enforce_immutable()

By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.

Examples

>>> from dffml import config
>>>
>>> @config
... class MyConfig:
...     C: int
>>>
>>> config = MyConfig(C=2)
>>> with config.no_enforce_immutable():
...     config.C = 1

class dffml.df.memory.MemoryRedundancyCheckerContext(config: dffml.base.BaseConfig, parent: dffml.df.memory.MemoryRedundancyChecker)

async unique(operation: dffml.df.types.Operation, parameter_set: dffml.df.base.BaseParameterSet) → str

SHA384 hash of the parameter set context handle as a string, the operation.instance_name, and the sorted list of input uuids.

class dffml.df.memory.NotificationSet

Set which can notifies user when it was added to (FIFO)