API

This part of the documentation covers all the interfaces of Mara Storage. For parts where the package depends on external libraries, we document the most important right here and provide links to the canonical documentation.

Storages

Provides a generic configuration interface for storages.

mara_storage.storages.storage(alias) → mara_storage.storages.Storage

Returns a storage configuration by alias

Storage client

Provides a generic storage client.

class mara_storage.client.StorageClient(storage: Union[str, mara_storage.storages.Storage])

A base class for a storage client

iterate_files(file_pattern: str) → Iterator[str]

Iterates over files on on a storage

Parameters

file_pattern – the file pattern, e.g. ‘subfolder/*.csv’

last_modification_timestamp(path: str) → datetime.datetime

Returns the last modification timestamp for a object (path or file) on a storage

File compression

Adds support to compress/uncompress file compressions while reading or writing (blob) data.

enum mara_storage.compression.Compression(value)

Different compression formats that are understood by file readers

Valid values are as follows:

NONE = <Compression.NONE: 'none'>
GZIP = <Compression.GZIP: 'gzip'>
TAR_GZIP = <Compression.TAR_GZIP: 'tar.gzip'>
ZIP = <Compression.ZIP: 'zip'>
mara_storage.compression.file_extension(compression: mara_storage.compression.Compression) → str

Gives the file extension for the compression

mara_storage.compression.compressor(compression: mara_storage.compression.Compression) → str

Maps compression methods to command line programs that can pack files. The list of files which shall be compressed must be given after the command. The compressed file will be send to stdout

Example

command = f’{compressor(Compression.GZIP)} my_file_to_compress.txt > my_compressed_file.txt.gz’

Parameters

compression – the compression to be used

Returns

The compress command without the files to be compressed

mara_storage.compression.uncompressor(compression: mara_storage.compression.Compression) → str

Maps compression methods to command line programs that can unpack the respective files

Shell commands

Functions to create shell commands to interact with a storage.

mara_storage.shell.read_file_command(storage: object, file_name: str, compression: mara_storage.compression.Compression = Compression.NONE) → str

Creates a shell command that reads a file and sends it to stdout

Parameters

  • storage – The storage where the file is stored

  • file_name – The file name within the storage

  • compression – The compression to be used to uncompress the file

Returns

A shell command string

mara_storage.shell.write_file_command(storage: object, file_name: str, compression: mara_storage.compression.Compression = Compression.NONE) → str

Creates a shell command that writes a file content from stdin

Parameters

  • storage – The storage where the file will be stored

  • file_name – The file name within the storage

  • compression – The compression to be used to compress the file

Returns

A shell command string

mara_storage.shell.delete_file_command(storage: object, file_name: str, force: bool = True, recursive: bool = False) → str

Creates a shell command that deletes a file on a storage

When the file does not exist, the command should silently end without error (exit code 0)

Parameters

  • storage – The storage where the file is stored

  • file_name – The file name within the storage

  • force – If True, the command will silently end without error (exit code 0) when the file does not exist

  • recursive – The folder with content will be deleted

Returns

A shell command string