Maintainers
Pricing
Enterprise
More
More
Documentation
Try Live
Rankings
Install
Blog
Docs7
Ask
New
Add Docs
RubyMarshal
https://rubymarshal.readthedocs.io/en/latest/
Admin
RubyMarshal is a Python library for reading and writing Ruby-marshalled data structures.
Tokens:
5,876
Snippets:
37
Trust Score:
9.6
Update:
1 month ago
Context
Chat
Benchmark
Latest
Show doc for...
Code
Info
Show Results
Tokens:
Raw
Copy
Link
Tokens:
Raw
Copy
Link
### Load and write Ruby-marshalled data with rubymarshal Source: https://rubymarshal.readthedocs.io/en/latest/_sources/index.txt Shows how to use rubymarshal.reader.load and rubymarshal.writer.write to read from and write to a file, and how to use loads and writes for in-memory operations. The example includes opening files in binary mode and demonstrates round-tripping a value. ```python from rubymarshal.reader import loads, load from rubymarshal.writer import writes, write with open('my_file', 'rb') as fd: content = load(fd) with open('my_file', 'wb') as fd: write(fd, my_object) loads(b"\x04\bi\xfe\x00\xff") writes(-256) ``` -------------------------------- ### Install rubymarshal with pip Source: https://rubymarshal.readthedocs.io/en/latest/_sources/index.txt Installs the RubyMarshal library using pip. ```bash pip install rubymarshal ``` -------------------------------- ### rubymarshal.writer module imports and constants with Writer class initialization Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Top of the source file with all imports and the Writer class definition, including the __init__ method that initializes symbol/object trackers and sets the output file descriptor. ```python # -*- coding: utf-8 -*- from __future__ import division, unicode_literals import re import io import math from rubymarshal.classes import Symbol, UsrMarshal from rubymarshal.constants import TYPE_BIGNUM, TYPE_STRING, TYPE_REGEXP, TYPE_ARRAY, TYPE_HASH, TYPE_USRMARSHAL, TYPE_NIL, TYPE_TRUE, TYPE_FALSE, \ TYPE_IVAR, TYPE_LINK, TYPE_SYMLINK, TYPE_SYMBOL, TYPE_FIXNUM from rubymarshal.constants import TYPE_FLOAT from rubymarshal.utils import write_ushort, write_sbyte, write_ubyte, integer_types, binary_type, text_type __author__ = 'Matthieu Gallet' re_class = re.compile('').__class__ simple_float_re = re.compile(r'^\d+\.\d*0+$') [docs]class Writer(object): def __init__(self, fd): self.symbols = {} self.objects = {} self.fd = fd ``` ```python [docs] def write(self, obj): if obj is None: self.fd.write(TYPE_NIL) elif obj is False: self.fd.write(TYPE_FALSE) elif obj is True: self.fd.write(TYPE_TRUE) elif isinstance(obj, int) or isinstance(obj, integer_types[-1]): if obj.bit_length() <= 5 * 8: self.fd.write(TYPE_FIXNUM) # noinspection PyTypeChecker self.write_long(obj) else: self.fd.write(TYPE_BIGNUM) if obj < 0: self.fd.write(b'-') else: self.fd.write(b'+') obj = abs(obj) size = int(math.ceil(obj.bit_length() / 16.)) self.write_long(size) for i in range(size): self.write_short(obj % 65536) obj //= 65536 elif isinstance(obj, Symbol): if obj.name in self.symbols: self.fd.write(TYPE_SYMLINK) self.write_long(self.symbols[obj.name]) else: self.fd.write(TYPE_SYMBOL) symbol_index = len(self.symbols) self.symbols[obj.name] = symbol_index encoded = obj.name.encode('utf-8') self.write_long(len(encoded)) self.fd.write(encoded) elif isinstance(obj, list): if self.must_write(obj): self.fd.write(TYPE_ARRAY) self.write_long(len(obj)) for x in obj: self.write(x) elif isinstance(obj, dict): if self.must_write(obj): self.fd.write(TYPE_HASH) self.write_long(len(obj)) for key, value in obj.items(): self.write(key) self.write(value) elif isinstance(obj, binary_type): self.fd.write(TYPE_IVAR) self.fd.write(TYPE_STRING) self.write_long(len(obj)) self.fd.write(obj) self.write_long(1) self.write(Symbol('E')) self.write(False) elif isinstance(obj, text_type): obj = obj.encode('utf-8') self.fd.write(TYPE_IVAR) self.fd.write(TYPE_STRING) self.write_long(len(obj)) self.fd.write(obj) self.write_long(1) self.write(Symbol('E')) self.write(True) elif isinstance(obj, float): obj = '%.20g' % obj if simple_float_re.match(obj): while obj.endswith('0'): obj = obj[:-1] obj = obj.encode('utf-8') self.fd.write(TYPE_FLOAT) self.write_long(len(obj)) self.fd.write(obj) elif isinstance(obj, re_class): flags = 0 if obj.flags & re.IGNORECASE: flags += 1 if obj.flags & re.DOTALL: flags += 4 self.fd.write(TYPE_IVAR) self.fd.write(TYPE_REGEXP) pattern = obj.pattern.encode('utf-8') self.write_long(len(pattern)) self.fd.write(pattern) write_ubyte(self.fd, flags) self.write_long(1) self.write(Symbol('E')) self.write(False) elif isinstance(obj, UsrMarshal): if self.must_write(obj): self.fd.write(TYPE_USRMARSHAL) self.write(Symbol(obj.cls)) self.write(obj.values) else: raise ValueError(obj) ``` ```python [docs] def write_short(self, obj): write_ushort(self.fd, obj) ``` -------------------------------- ### write: Serialize object to file descriptor Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Writes the Ruby Marshal header (\x04\x08) and then serializes the object using a Writer instance. ```python def write(fd, obj): fd.write(b'\x04\x08') writer = Writer(fd) writer.write(obj) ``` -------------------------------- ### rubymarshal.classes.Module Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a Ruby module. Constructor takes the class and values. ```APIDOC ## Module ### Description Represents a Ruby module. ### Constructor `Module(cls, values)` ### Parameters - **cls** - The class of the module. - **values** - The values of the module. ``` -------------------------------- ### Reader.read_short method implementation Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html The `read_short` method reads an unsigned short integer from the file descriptor using the `read_ushort` helper function. ```python [docs] def read_short(self): return read_ushort(self.fd) ``` -------------------------------- ### write(fd, obj) Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/writer.html Writes the given object to the file-like object fd in Marshal format. ```APIDOC ## write(fd, obj) ### Description Writes the given object to the file-like object `fd` in Marshal format. ### Parameters - `fd` (file-like object) - Required - The file-like object to write to. - `obj` (object) - Required - The object to serialize. ``` -------------------------------- ### rubymarshal.classes.Object Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a Ruby object. Constructor takes the class, values, and additional keyword arguments. ```APIDOC ## Object ### Description Represents a Ruby object. ### Constructor `Object(cls, values, **kwargs)` ### Parameters - **cls** - The class of the object. - **values** - The values of the object. - **kwargs** - Additional keyword arguments. ``` -------------------------------- ### Symbol class definition Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/classes.html Defines the Symbol class with interning via __registered_symbols__, and implements __new__, __init__, __hash__, __repr__, __str__, and __unicode__ methods. ```python [docs]class Symbol(object): __registered_symbols__ = {} def __new__(cls, name): if name in cls.__registered_symbols__: return cls.__registered_symbols__[name] return super(Symbol, cls).__new__(cls) def __init__(self, name): self.name = name self.__registered_symbols__[name] = self def __hash__(self): return hash('<<<:%s:>>>' % self.name) def __repr__(self): return 'Symbol("%s")' % self.name def __str__(self): return ':%s' % self.name def __unicode__(self): return ':%s' % self.name ``` -------------------------------- ### read_symreal method Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Reads a real symbol by reading its size, then the symbol bytes, decoding as UTF-8, and appending to the symbols list. ```python [docs] def read_symreal(self): size = self.read_long() result = self.fd.read(size) result = Symbol(result.decode('utf-8')) self.symbols.append(result) return result ``` -------------------------------- ### must_write: Track object references for links Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Checks if an object has already been written to avoid duplication. If it has, writes a link reference and returns False; otherwise registers the object and returns True. ```python def must_write(self, obj): if id(obj) in self.objects: self.fd.write(TYPE_LINK) self.write_long(self.objects[id(obj)]) return False else: link_index = len(self.objects) self.objects[id(obj)] = link_index return True ``` -------------------------------- ### writes(obj) Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/writer.html Serializes the given object to a bytes string in Marshal format. ```APIDOC ## writes(obj) ### Description Serializes the given object to a bytes string in Marshal format. ### Parameters - `obj` (object) - Required - The object to serialize. ### Returns - bytes - The serialized Marshal data. ``` -------------------------------- ### rubymarshal.classes.RubyClass Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a Ruby class. Constructor takes the class and values. ```APIDOC ## RubyClass ### Description Represents a Ruby class. ### Constructor `RubyClass(cls, values)` ### Parameters - **cls** - The class of the Ruby class. - **values** - The values of the Ruby class. ``` -------------------------------- ### load function Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Loads a Ruby Marshal object from a file-like object. Asserts the first two bytes are the Marshal header (0x04 0x08) and uses a Reader to parse the rest. ```python [docs]def load(fd): assert fd.read(1) == b'\x04' assert fd.read(1) == b'\x08' loader = Reader(fd) return loader.read() ``` -------------------------------- ### load function Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/reader.html The load function loads Ruby Marshal data from a file-like object. ```APIDOC ## load ### Description Loads Ruby Marshal data from a file-like object. ### Parameters - **fd** (file-like object) - Required - The file-like object to read from. ### Returns The deserialized Ruby object. ``` -------------------------------- ### rubymarshal.classes.Extended Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents an extended Ruby object. Constructor takes the class and values. ```APIDOC ## Extended ### Description Represents an extended Ruby object. ### Constructor `Extended(cls, values)` ### Parameters - **cls** - The class of the object. - **values** - The values of the object. ``` -------------------------------- ### read_symlink method Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Reads a symbol link by reading a long integer ID and returning the corresponding symbol from the symbols list. ```python [docs] def read_symlink(self): symlink_id = self.read_long() return self.symbols[symlink_id] ``` -------------------------------- ### rubymarshal.classes.UsrMarshal Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a user-marshaled Ruby object. Constructor takes the class and values. ```APIDOC ## UsrMarshal ### Description Represents a user-marshaled Ruby object. ### Constructor `UsrMarshal(cls, values)` ### Parameters - **cls** - The class of the object. - **values** - The values of the object. ``` -------------------------------- ### loads function Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Loads a Ruby Marshal object from a bytes object by wrapping it in a BytesIO and calling load. ```python [docs]def loads(byte_text): return load(io.BytesIO(byte_text)) ``` -------------------------------- ### rubymarshal.classes.Symbol Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a Ruby symbol. Constructor takes the name. ```APIDOC ## Symbol ### Description Represents a Ruby symbol. ### Constructor `Symbol(name)` ### Parameters - **name** - The name of the symbol. ``` -------------------------------- ### loads function Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/reader.html The loads function loads Ruby Marshal data from a byte string. ```APIDOC ## loads ### Description Loads Ruby Marshal data from a byte string. ### Parameters - **byte_text** (bytes) - Required - The byte string containing the Marshal data. ### Returns The deserialized Ruby object. ``` -------------------------------- ### Writer.write(obj) Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Serializes a Python object into Ruby Marshal format by writing bytes to the internal file descriptor. Handles various Python types by converting them to corresponding Ruby type tags and maintaining symbol and object tables for references. ```APIDOC # Writer.write(obj) ## Description Serializes a Python object into Ruby Marshal format and writes the resulting bytes to the internal file descriptor. It handles the conversion of Python types to Ruby equivalents, including special handling for symbols, arrays, hashes, strings, regexes, floats, and user-defined Marshal classes. ## Parameters ### obj - **Type**: Any (Python object) - **Required**: Yes - **Description**: The Python object to serialize. Supported types include None, bool, int, str, bytes, list, dict, float, compiled regex, Symbol, UsrMarshal, and user-defined classes (if explicitly a subclass of UsrMarshal). ## Behavior - **None** → Ruby nil - **False** → Ruby false - **True** → Ruby true - **int** (within 5 bytes) → Fixnum (TYPE_FIXNUM) - **int** (large) → Bignum (TYPE_BIGNUM) - **Symbol** → symbol or symlink - **list** → Array (TYPE_ARRAY) - **dict** → Hash (TYPE_HASH) - **bytes/str** → string with encoding flag (TYPE_IVAR + TYPE_STRING) - **float** → Float (TYPE_FLOAT) - **compiled regex** → Regexp (TYPE_IVAR + TYPE_REGEXP) - **UsrMarshal** instance → UsrMarshal (TYPE_USRMARSHAL) ## Throws - ValueError: If the object type is not supported. ## Example ```python from rubymarshal.writer import Writer import io # Sample usage fd = io.BytesIO() writer = Writer(fd) writer.write([{'a': 1}, 2.5, 'hello', None]) ``` ``` -------------------------------- ### rubymarshal.classes.UserDef Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/classes.html Represents a user-defined Ruby object. Constructor takes the class and values. ```APIDOC ## UserDef ### Description Represents a user-defined Ruby object. ### Constructor `UserDef(cls, values)` ### Parameters - **cls** - The class of the object. - **values** - The values of the object. ``` -------------------------------- ### Reader.read method implementation Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html The `read` method recursively parses a Ruby Marshal stream. It handles all token types defined by the Marshal format, including TYPE_NIL, TYPE_TRUE, TYPE_FALSE, TYPE_IVAR, TYPE_STRING, TYPE_SYMBOL, TYPE_FIXNUM, TYPE_ARRAY, TYPE_HASH, TYPE_FLOAT, TYPE_BIGNUM, TYPE_REGEXP, TYPE_USRMARSHAL, TYPE_SYMLINK, TYPE_LINK, TYPE_USERDEF, TYPE_MODULE, TYPE_OBJECT, and TYPE_EXTENDED. It maintains an object table for link references and handles encoding for strings and regexps. ```python [docs] def read(self, token=None, ivar=False): if token is None: token = self.fd.read(1) object_index = None if token in (TYPE_CLASS, TYPE_MODULE, TYPE_FLOAT, TYPE_BIGNUM, TYPE_STRING, TYPE_REGEXP, TYPE_ARRAY, TYPE_HASH, TYPE_STRUCT, TYPE_OBJECT, TYPE_DATA, TYPE_USRMARSHAL): self.objects.append(None) object_index = len(self.objects) if token == TYPE_NIL: result = None elif token == TYPE_TRUE: result = True elif token == TYPE_FALSE: result = False elif token == TYPE_IVAR: sub_token = self.fd.read(1) content = self.read(sub_token, ivar=True) flags = None if sub_token != TYPE_STRING: print(token, sub_token, '-----') if sub_token == TYPE_REGEXP: options = ord(self.fd.read(1)) flags = 0 if options & 1: flags |= re.IGNORECASE if options & 4: flags |= re.DOTALL if sub_token in (TYPE_STRING, TYPE_REGEXP, TYPE_USERDEF): encoding = 'latin1' attr_count = self.read_long() attrs = {} for x in range(attr_count): attr_name = self.read() attr_value = self.read() if attr_name == Symbol('E') and attr_value is True: encoding = 'utf-8' elif attr_name == Symbol('encoding'): encoding = attr_value.decode('utf-8') attrs[attr_name] = attr_value if sub_token in (TYPE_STRING, TYPE_REGEXP): content = content.decode(encoding) else: raise ValueError if sub_token == TYPE_REGEXP: content = re.compile(content, flags) result = content elif token == TYPE_STRING: size = self.read_long() result = self.fd.read(size) elif token == TYPE_SYMBOL: result = self.read_symreal() elif token == TYPE_FIXNUM: result = self.read_long() elif token == TYPE_ARRAY: num_elements = self.read_long() # noinspection PyUnusedLocal result = [self.read() for x in range(num_elements)] elif token == TYPE_HASH: num_elements = self.read_long() result = {} for x in range(num_elements): key = self.read() value = self.read() result[key] = value result = result elif token == TYPE_FLOAT: size = self.read_long() result = float(self.fd.read(size).decode('utf-8')) elif token == TYPE_BIGNUM: sign = 1 if self.fd.read(1) == b'+' else -1 num_elements = self.read_long() result = 0 factor = 1 for x in range(num_elements): result += (self.read_short() * factor) factor *= 2 ** 16 result *= sign elif token == TYPE_REGEXP: size = self.read_long() result = self.fd.read(size) elif token == TYPE_USRMARSHAL: class_name = self.read() attr_list = self.read() cls = self.usr_marshal_mapping.get(class_name, UsrMarshal) result = cls(class_name, attr_list) elif token == TYPE_SYMLINK: result = self.read_symlink() elif token == TYPE_LINK: link_id = self.read_long() result = self.objects[link_id] elif token == TYPE_USERDEF: class_name = self.read() data = self.read(TYPE_STRING) if not ivar: data = loads(data) result = UserDef(class_name, data) print(result) elif token == TYPE_MODULE: module_name = self.read() result = Module(module_name, None) elif token == TYPE_OBJECT: class_name = self.read() data = self.read(TYPE_STRING) # attr_count = self.read_long() # # noinspection PyUnusedLocal # attrs = {self.read(): self.read() for x in range(attr_count)} if not ivar: data = self.read() result = Object(class_name, data) print(result) elif token == TYPE_EXTENDED: class_name = self.read(TYPE_STRING) print(class_name) result = Extended(class_name, None) else: raise ValueError('token %s is not recognized' % token) if object_index is not None: self.objects[object_index - 1] = result # print('end', result, object_index, self.objects) return result ``` -------------------------------- ### writes: Serialize object to bytes Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Serializes an object to a bytes object using an in-memory buffer. ```python def writes(obj): fd = io.BytesIO() write(fd, obj) return fd.getvalue() ``` -------------------------------- ### Writer class Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/writer.html The Writer class is used to write Ruby objects to a file-like object in Marshal format. ```APIDOC ## Writer ### Description The `Writer` class is used to write Ruby objects to a file-like object in Marshal format. ### Constructor `Writer(fd)` ### Methods - `must_write(obj)` - `write(obj)` - `write_long(obj)` - `write_short(obj)` ``` -------------------------------- ### RubyClass and subclasses definition Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/classes.html Defines the RubyClass base class with __init__, __eq__, __hash__, __repr__, __str__, and __unicode__ methods. Subclasses UsrMarshal, UserDef, Extended, Module, and Object inherit from RubyClass; Object adds a kwargs attribute. ```python # -*- coding: utf-8 -*- from __future__ import unicode_literals __author__ = 'Matthieu Gallet' [docs]class RubyClass(object): def __init__(self, cls, values): if isinstance(cls, Symbol): cls = cls.name self.cls = cls self.values = values def __eq__(self, other): return isinstance(other, self.__class__) and self.cls == other.cls and self.values == other.values def __hash__(self): return hash(hash(self.cls) + hash(self.values)) def __repr__(self): return '%s:%s(%r)' % (self.__class__.__name__, self.cls, self.values) def __str__(self): return '%s:%s(%r)' % (self.__class__.__name__, self.cls, self.values) def __unicode__(self): return '%s:%s(%r)' % (self.__class__.__name__, self.cls, self.values) [docs]class UsrMarshal(RubyClass): pass [docs]class UserDef(RubyClass): pass [docs]class Extended(RubyClass): pass [docs]class Module(RubyClass): pass [docs]class Object(RubyClass): def __init__(self, cls, values, **kwargs): self.kwargs = kwargs super(Object, self).__init__(cls, values) ``` -------------------------------- ### read_symbol method Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Reads a symbol from the stream, handling encoding markers and symbol links. Raises ValueError if a symbol link appears with an encoding marker. ```python [docs] def read_symbol(self): ivar = 0 while True: token = self.fd.read(1) if token == TYPE_IVAR: ivar = 1 elif token == TYPE_SYMBOL: return self.read_symreal() elif token == TYPE_SYMLINK: if ivar: raise ValueError('dump format error (symlink with encoding)') return self.read_symlink() ``` -------------------------------- ### Reader class Source: https://rubymarshal.readthedocs.io/en/latest/api/rubymarshal/reader.html The Reader class is used to read Ruby Marshal data from a file-like object. It supports custom marshal and user-defined mappings. ```APIDOC ## Reader ### Description Reader is a class for reading Ruby Marshal data from a file-like object. ### Constructor - **fd** (file-like object) - Required - The file-like object to read from. - **usr_marshal_mapping** (optional) - Custom marshal mapping. - **userdef_mapping** (optional) - Custom user-defined mapping. ### Methods - **read(token=None, ivar=False)** - Reads the next object from the stream. - **read_long()** - Reads a long integer. - **read_short()** - Reads a short integer. - **read_symbol()** - Reads a symbol. - **read_symlink()** - Reads a symbol link. - **read_symreal()** - Reads a real symbol. ``` -------------------------------- ### write_long: Serialize integer to Ruby Marshal Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/writer.html Serializes an integer to the Ruby Marshal format. Handles zero, small positive/negative values, and larger values with a variable-length encoding. Raises ValueError if the integer is too large (more than 5 bytes). ```python def write_long(self, obj): if obj == 0: self.fd.write(b'\0') elif 0 < obj < 123: write_sbyte(self.fd, obj + 5) elif -124 < obj < 0: write_sbyte(self.fd, obj - 5) else: size = int(math.ceil(obj.bit_length() / 8.)) if size > 5: raise ValueError('%d too long for serialization' % obj) original_obj = obj factor = 256 ** size if obj < 0 and obj == -factor: size -= 1 obj += (factor / 256) elif obj < 0: obj += factor sign = int(math.copysign(size, original_obj)) write_sbyte(self.fd, sign) for i in range(size): write_ubyte(self.fd, obj % 256) obj //= 256 ``` -------------------------------- ### read_long method Source: https://rubymarshal.readthedocs.io/en/latest/_modules/rubymarshal/reader.html Reads a Ruby Marshal long integer from the file descriptor. Handles small integers directly and larger ones via byte-by-byte accumulation. Returns the integer value. ```python [docs] def read_long(self): l = read_sbyte(self.fd) if l == 0: return 0 if 5 < l < 128: return l - 5 elif -129 < l < -5: return l + 5 result = 0 factor = 1 for s in range(abs(l)): result += (read_ubyte(self.fd) * factor) factor *= 256 if l < 0: result = result - factor return result ```