evox.vis_tools.exv
#
This module helps serialize data to EvoXVision storage format (exv).
Magic Number |
Header Length |
Metadata |
Initial Iteration Data |
Binary Data |
---|---|---|---|---|
0x65787631 (4 bytes) |
u32 (4 bytes) |
JSON encoded (n bytes) |
binary data |
binary data |
The numbers are stored in little-endian format. The metadata is a JSON utf-8 encoded string, which contains the schema of the binary data. The format of the metadata is as follows:
{
"version": "v1",
"n_objs": "<number>",
"initial_iteration": {
"population_size": "<number>",
"chunk_size": "<number>",
"fields": [
{
"name": "<field name>",
"type": "<type>",
"size": "<number>",
"offset": "<number>",
"shape": ["<number>"]
}
]
},
"rest_iterations": {
"population_size": "<number>",
"chunk_size": "<number>",
"fields": [
{
"name": "<field name>",
"type": "<type>",
"size": "<number>",
"offset": "<number>",
"shape": ["<number>"]
}
]
}
}
where
“u8”, “u16”, “u32”, “u64”,
“i16”, “i32”, “i64”,
“f16”, “f32”, “f64” The size and offset are in bytes.
Note
The magic number is used to identify the file format. 0x65787631 is the byte code for “exv1”. The binary data blob is a sequence of binary data chunks. In EvoX, the algorithm is allowed to have a different behavior in the first iteration (initialization phase), which can have a different chunk size than the rest of the iterations. Therefore it contains two different schemas for the initial iteration and the rest of the iterations.
Module Contents#
Classes#
EvoXVisionAdapter is a class that streams evolutionary optimization data to an exv file. The exv file format is a binary format that created specifically for the evolutionary optimization data. The format is designed to be efficient for both stream reading and writing data, while being able to randomly access data at any iteration. |
Functions#
Takes the input of the populaton and fitness from the first two iterations, and returns the schema for exv file format. |
API#
- evox.vis_tools.exv._get_data_type(dtype)#
- evox.vis_tools.exv.new_exv_metadata(population1: numpy.ndarray, population2: numpy.ndarray, fitness1: numpy.ndarray, fitness2: numpy.ndarray)#
Takes the input of the populaton and fitness from the first two iterations, and returns the schema for exv file format.
- class evox.vis_tools.exv.EvoXVisionAdapter(file_path: Union[str, pathlib.Path], buffering: int = 0)#
EvoXVisionAdapter is a class that streams evolutionary optimization data to an exv file. The exv file format is a binary format that created specifically for the evolutionary optimization data. The format is designed to be efficient for both stream reading and writing data, while being able to randomly access data at any iteration.
Initialization
Create a new EvoXVisionAdapter instance, which writes data to an exv file. To automatically inference the data schema, the EvoXVisionAdapter requires 2 iterations of data, therefore it will only start to write data after the 2 iterations of the optimization loop are completed.
- Parameters:
file_path – The path to the exv file
buffering – The buffer size to use for file operations, passed directly to the
open()
function. The default is0
, which disables buffering (unbuffered mode).
- _write_magic_number()#
- _write_metedata(metadata)#
- set_metadata(metadata)#
- write_header()#
Write the header of the exv file.
- write(*fields)#
Stream data to the exv file. Depending on the
buffering
parameter, the data may not be written immediately.
- flush()#
Flush the internal buffer to the file.