evox.vis_tools.exv#

This module helps serialize data to EvoXVision storage format (exv).

Magic Number

Header Length

Metadata

Initial Iteration Data

Binary Data

0x65787631 (4 bytes)

u32 (4 bytes)

JSON encoded (n bytes)

binary data

binary data

The numbers are stored in little-endian format. The metadata is a JSON utf-8 encoded string, which contains the schema of the binary data. The format of the metadata is as follows:

{
    "version": "v1",
    "n_objs": "<number>",
    "initial_iteration": {
        "population_size": "<number>",
        "chunk_size": "<number>",
        "fields": [
            {
                "name": "<field name>",
                "type": "<type>",
                "size": "<number>",
                "offset": "<number>",
                "shape": ["<number>"]
            }
        ]
    },
    "rest_iterations": {
        "population_size": "<number>",
        "chunk_size": "<number>",
        "fields": [
            {
                "name": "<field name>",
                "type": "<type>",
                "size": "<number>",
                "offset": "<number>",
                "shape": ["<number>"]
            }
        ]
    }
}

where represents the data type of the field, available types are:

  • “u8”, “u16”, “u32”, “u64”,

  • “i16”, “i32”, “i64”,

  • “f16”, “f32”, “f64” The size and offset are in bytes.

Note

The magic number is used to identify the file format. 0x65787631 is the byte code for “exv1”. The binary data blob is a sequence of binary data chunks. In EvoX, the algorithm is allowed to have a different behavior in the first iteration (initialization phase), which can have a different chunk size than the rest of the iterations. Therefore it contains two different schemas for the initial iteration and the rest of the iterations.

Module Contents#

Classes#

EvoXVisionAdapter

EvoXVisionAdapter is a class that streams evolutionary optimization data to an exv file. The exv file format is a binary format that created specifically for the evolutionary optimization data. The format is designed to be efficient for both stream reading and writing data, while being able to randomly access data at any iteration.

Functions#

_get_data_type

new_exv_metadata

Takes the input of the populaton and fitness from the first two iterations, and returns the schema for exv file format.

API#

evox.vis_tools.exv._get_data_type(dtype)#
evox.vis_tools.exv.new_exv_metadata(population1: numpy.ndarray, population2: numpy.ndarray, fitness1: numpy.ndarray, fitness2: numpy.ndarray)#

Takes the input of the populaton and fitness from the first two iterations, and returns the schema for exv file format.

class evox.vis_tools.exv.EvoXVisionAdapter(file_path: Union[str, pathlib.Path], buffering: int = 0)#

EvoXVisionAdapter is a class that streams evolutionary optimization data to an exv file. The exv file format is a binary format that created specifically for the evolutionary optimization data. The format is designed to be efficient for both stream reading and writing data, while being able to randomly access data at any iteration.

Initialization

Create a new EvoXVisionAdapter instance, which writes data to an exv file. To automatically inference the data schema, the EvoXVisionAdapter requires 2 iterations of data, therefore it will only start to write data after the 2 iterations of the optimization loop are completed.

Parameters:
  • file_path – The path to the exv file

  • buffering – The buffer size to use for file operations, passed directly to the open() function. The default is 0, which disables buffering (unbuffered mode).

_write_magic_number()#
_write_metedata(metadata)#
set_metadata(metadata)#
write_header()#

Write the header of the exv file.

write(*fields)#

Stream data to the exv file. Depending on the buffering parameter, the data may not be written immediately.

flush()#

Flush the internal buffer to the file.