Selection @pythonetc, April 2019

Selection @pythonetc, April 2019




This is the tenth collection of tips on Python and programming from my author’s @pythonetc channel.

Previous Collections .





Storing and sending objects over the network in the form of bytes is a very big topic. For these purposes, Python usually uses a number of tools, let's discuss their advantages and disadvantages.

As an example, I will try to serialize a Cities object that contains City objects in a specific order. Four approaches can be taken:

1. JSON. Human-readable, easy to use, but consumes a lot of memory. The same is true of the YAML and XML formats.

  class City:
  def to_dict (self):
  return dict (
  name = self._name,
  country = self._country,
  lon = self._lon,
  lat = self._lat,
  )

 class Cities:
  def __init __ (self, cities):
  self._cities = cities

  def to_json (self):
  return json.dumps ([
  c.to_dict () for c in self._cities
  ]). encode ('utf8')  

2. Pickle. This native Python tool, customizable, consumes less memory than JSON. Disadvantage: Python must be used to extract data.

  class Cities:
  def pickle (self):
  return pickle.dumps (self)  

3. Protobuf (and other binary serializers, for example, msgpack). It consumes even less memory, can be used from any programming language, but requires writing an explicit scheme:

  syntax = "proto2";


 message City {
  required string name = 1;
  required string country = 2;
  required float lon = 3;
  required float lat = 4;
 }

 message Cities {
  repeated City cities = 1;
 }

 class City:
  def to_protobuf (self):
  result = city_pb2.City ()
  result.name = self._name
  result.country = self._country
  result.lon = self._lon
  result.lat = self._lat

  return result

 class Cities:
  def to_protobuf (self):
  result = city_pb2.Cities ()
  result.cities.extend ([
  c.to_protobuf () for c in self._cities
  ])

  return result  

4. Manually. You can manually package and unpack data using the struct module. This way you can achieve the lowest possible memory consumption, however sometimes it is better to use protobuf , because it supports versioning and explicit schemes.

  class City:
  def to_bytes (self):
  name_encoded = self._name.encode ('utf8')
  name_length = len (name_encoded)

  country_encoded = self._country.encode ('utf8')
  country_length = len (country_encoded)

  return struct.pack (
  'BsBsff',
  name_length, name_encoded,
  country_length, country_encoded,
  self._lon, self._lat,
  )

 class Cities:
  def to_bytes (self):
  return b ''. join (
  c.to_bytes () for c in self._cities
  )  





If the function argument has a default value of None and is annotated as T , then mypy will automatically read it Optional [T] (i.e. Union [T, None] ).

This doesn't work with other types, so you can't write something like f (x: A = B ()) . Also this trick does not work with assigning a variable: a: A = None will lead to an error.

  def f (x: int = None):
  reveal_type (x)

 def g (y: int = 'x'):
  reveal_type (y)

 z: int = None
 reveal_type (z)

 $ mypy test.py
 test.py:2: error: Revealed type is 'Union [builtins.int, None]'
 test.py:4: error: Incompatible default for argument "y" (default has type "str", argument has type "int")
 test.py.22: error: Revealed type is 'builtins.int'
 test.py: 7: error: Incompatible types in assignment (expression has type "None", variable has type "int")
 test.py:8: error: Revealed type is 'builtins.int'  

***

In Python 3, when you exit the except block, the variables that store the caught exceptions are deleted from locals () , even if they already existed:

  & gt; & gt; & gt;  e = 2
 & gt; & gt; & gt;  try:
 ... 1/0
 ... except Exception as e:
 ... pass
 ...
 & gt; & gt; & gt;  e
 Traceback (most recent call last):
  File "& lt; stdin & gt;", line 1, in & lt; module & gt;
 NameError: name 'e' is not defined  

If you want to keep the reference to the exception, you need to use another variable:

  & gt; & gt; & gt;  error = None
 & gt; & gt; & gt;  try:
 ... 1/0
 ... except Exception as e:
 ... error = e
 ...
 & gt; & gt; & gt;  error
 ZeroDivisionError ('division by zero',)  

In Python 2, however, this does not happen.




You can easily make your own pypi repository. It allows you to release packages inside your project and install them using pip , as if they were regular packages.

It is important to note that you do not need to install any special software, you can use a regular HTTP server. Here's how it works for me.

Take the primitive pythonetc package.

  setup.py:

 from setuptools import setup, find_packages

 setup (
  name = 'pythonetc',
  version = '1.0',
  packages = find_packages (),
 )

 pythonetc.py:

 def ping ():
  return 'pong'  

Let's release it to the ~/pypi directory:

  $ python setup.py sdist bdist_wheel
 ...
 $ mv dist ~/pypi/pythonetc  

And we will start providing the package from the pypi.pushtaev.ru domain using nginx:

  $ cat/etc/nginx/sites-enabled/pypi
 server {
  listen 80;
  server_name pypi.pushtaev.ru;
  root/home/vadim/pypi;

  index index.html index.htm index.nginx-debian.html;

  location/{
  autoindex on;
  try_files $ uri $ uri/= 404;
  }
 }  

Now the package can be installed:

  $ pip install -i http://pypi.pushtaev.ru --trusted-host pypi.pushtaev.ru pythonetc
 ...
 Collecting pythonetc
  Downloading http://pypi.pushtaev.ru/pythonetc/pythonetc-1.0-py3-none-any.whl
 Installing collected packages: pythonetc
 Successfully installed pythonetc-1.0
 $ python
 Python 3.7.0+ (heads/3.7: 0964aac, Mar 29 2019, 00:40:55)
 [GCC 4.9.2] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 & gt; & gt; & gt;  import pythonetc
 & gt; & gt; & gt;  pythonetc.ping ()
 'pong'  





Often you need to declare a dictionary with keys of the same name as local variables. For example:

  dict (
  context = context,
  mode = mode,
  action_type = action_type,
 )  

In ECMAScript for such cases there is even a special form of the literal object (called Object Literal Property Value Shorthand):

  & gt;  var a = 1;
 & lt;  undefined
 & gt;  var b = 2;
 & lt;  undefined
 & gt;  {a, b}
 & lt;  {a: 1, b: 2}  

You can create the same helper in Python (alas, it’s not at all as good as ECMAScript notation):

  def shorthand_dict (lcls, names):
  return {k: lcls [k] for k in names}

 context = dict (user_id = 42, user_ip = '1.2.3.4')
 mode = 'force'
 action_type = 7

 shorthand_dict (locals (), [
  'context',
  'mode',
  'action_type',
 ])  

You may ask, why pass locals () as a parameter? Can you get the locals of the caller in the callee? You can, but you'll have to use the inspect module:

  import inspect

 def shorthand_dict (names):
  lcls = inspect.currentframe (). f_back.f_locals
  return {k: lcls [k] for k in names}

 context = dict (user_id = 42, user_ip = '1.2.3.4')
 mode = 'force'
 action_type = 7

 shorthand_dict ([
  'context',
  'mode',
  'action_type',
 ])  

You can go further and apply this solution - https://github.com/alexmojaki/sorcery :

  from sorcery import dict_of
 dict_of (context, mode, action_type)
  

Source text: Selection @pythonetc, April 2019