Cannot serialize bytestring to JSON

1

Hi everyone. I hope you’re safe and healthy.

I built an RPC service based on the Nameko framework. The service class implements a method which takes a list of strings as an argument and returns a dictionary with each original string as a key and the string Huffman encoded as a value. The encoding works, but I cannot send the return value across the network because the encoded bytestrings cannot be serialized to JSON.

If I import the service class and test the methods using PyTest, this problem is not reproduced.

How can I return a Huffman encoded bytestring over a wire as JSON?

ENCODING METHOD:

 22     @rpc
 23     def huffman_encode_strings(self, items):
 24         """Build a dictionary of strings - the key being the original string,
 25         and the value being a (Huffman) encoded version of that string.
 26
 27         Parameters:
 28             items (list): a list of strings.
 29
 30         Returns:
 31             A dictionary of Huffman encoded strings with original strings as
 32             keys and encoded strings as values.
 33         """
 34         encoded_items = {}
 35         for item in items:
 36             item_codec = HuffmanCodec.from_data(item)
 37             encoded_item = item_codec.encode(item)
 38             encoded_items.update({item: encoded_item})
 39         return encoded_items

ERROR:

(rpc-microservice) MacBook-Pro:rpc-microservice sd$ nameko shell
Nameko Python 3.6.11 (default, Aug  7 2020, 03:28:31)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62)] shell on darwin
Broker: pyamqp://guest:guest@localhost
>>> service = n.rpc.acme_rpc_microservice
>>> strings = ["A man, a plan, a canal: Panama!"]
>>> ret = service.huffman_encode_strings(strings)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/sd/.pyenv/versions/3.6.11/envs/rpc-microservice/lib/python3.6/site-packages/nameko/rpc.py", line 373, in __call__
    return reply.result()
  File "/Users/sd/.pyenv/versions/3.6.11/envs/rpc-microservice/lib/python3.6/site-packages/nameko/rpc.py", line 331, in result
    raise deserialize(error)
nameko.exceptions.RemoteError: UnserializableValueError Unserializable value: `{'A man, a plan, a canal: Panama!': b'Y\xd6B<\xf6B\x1bi\xd42\xd7_\xfc'}`

TIA.

2

I think nameko supports simple data structures that you can easily json dump otherwise you can still handle json to string conversion yourself and return the string

1 Like
3

Thanks Evans. So I should be looking into giving Nameko a utf-8 value by coercing the output of my encoder?

4

With some advice I solved this by encoding bytestrings to latin1 before moving them “across the wire”.

Thanks + stay safe everyone.