rlgt.environments.graph_environment.GraphEnvironment

This abstract class encapsulates the concept of a reinforcement learning (RL) environment for graph theory applications. Such an environment is designed to address extremal problems in which a specified graph invariant is to be maximized over a finite family of fully colored k-edge-colored looped complete graphs.

States are represented as fixed-length numpy.ndarray vectors. Actions are represented as numpy.int32 integers taking values in the range from 0 to action_number - 1, where action_number denotes the total (finite) number of available actions. At each step, some actions may be unavailable, with the constraint that at least one action is always available in any non-terminal state.

For efficiency, the environment supports running multiple episodes in parallel. All episodes are guaranteed to terminate after a predetermined number of steps, regardless of whether the underlying task is episodic or continuing. Accordingly, batches of states are represented as two-dimensional numpy.ndarray matrices whose rows correspond to individual states, while batches of actions are represented as one-dimensional numpy.ndarray vectors of type numpy.int32.

Instead of returning rewards in the conventional RL sense, the environment returns values of a selected graph invariant associated with the underlying graphs corresponding to the newly reached states. The graph invariant is specified via a GraphInvariant function. If the sparse setting is enabled, graph invariant values are computed only for the final batch of actions, and None is returned at all preceding steps. Otherwise, the invariant values are computed after every batch of actions. In the latter case, the computation may be optimized by supplying a GraphInvariantDiff function, which specifies how graph invariant values change when the environment transitions from one batch of underlying graphs to another.

Conceptually, environments are divided into two types according to the nature of the underlying RL task: the continuing environments and the episodic environments. In continuing environments, the task has no terminal states in the usual RL sense, and the underlying graph associated with each state is guaranteed to be fully colored. In episodic environments, the underlying graph associated with a terminal state is guaranteed to be fully colored, but no such guarantee is made for the underlying graphs associated with non-terminal states.

Concrete subclasses must implement the following abstract properties:

1. state_length, which returns the length of the state vectors;
2. state_dtype, which returns the data type of the state vectors;
3. action_number, which returns the total number of available actions;
4. action_mask, which specifies which actions are currently available;
5. episode_length, which returns the predetermined length of each episode; and
6. is_continuing, which determines whether the environment is continuing or episodic.

Concrete subclasses must also implement the following abstract methods:

1. _initialize_batch, which initializes a batch of episodes;
2. _transition_batch, which applies a batch of actions to the current batch of states; and
3. state_batch_to_graph_batch, which extracts the underlying batch of graphs from a provided batch of states.