Introduction to

Advanced Python

@1995parham
python-logo
Summer 2017

In the memory of our friends who are not among us right now

Parham Alvani

email parham.alvani@gmail.com

home 1995parham.github.io

1995parham-logo

Who I am?

school BSc. Software Engineering @ Amirkabir University of Technology

school MSc. Computer Networks @ Amirkabir University of Technology

domain Since June 2015 (Ordibehesht 1394) in IoT/Platforms

ceit-logo aut-logo

Where do I work?

Bambil

Where do I work?

Nahal

Lian Smart Cloud based Monitoring

Why advanced python ?

Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.

Zen of Python (cont'd)

In the face of ambiguity, refuse the temptation to guess.
There should be one—and preferably only one—obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.

Zen of Python (cont'd)

If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea—let's do more of those!

Outline

Python Version

3.6

  • Formatted string literals
  • Underscores in Numeric Literals
  • Syntax for variable annotations
  • Simpler customization of class creation
  • Descriptor Protocol Enhancement
  • asyncio

Trust in docs.python.org

book Python Documentation

Python vs Nodejs

I love them both

favorite

Knowledge is required to understand examples

  • HTTP Protocol
  • Networking Stack
  • Basis of algorithms

Github + Laptop

==

You have the codes

What is Python !?

What is Python !?

Python is a high-level, interpreted and object-oriented scripting language.

Python is designed to be highly readable.

Swap


            a = a + b;
            b = a - b;
            a = a - b;
            

            a, b = b, a
            

Alternative Python Implementations

  • Python (nicknamed CPython)
  • IronPython (Python running on .NET)
  • Jython (Python running on the Java Virtual Machine)
  • PyPy (A fast python implementation with a JIT compiler)
  • Stackless Python (Branch of CPython supporting microthreads)
  • MicroPython (Python running on micro controllers)

Stackless

Stackless Python is an enhanced version of the Python programming language. It allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads.

  • Microthreads
  • Channels
  • Scheduling
  • Serialization

Python Terminology

PyPi

the Python Package Index

PyPA

The Python Packaging Authority (PyPA) is a working group that maintains many of the relevant projects in Python packaging

pip

The PyPA recommended tool for installing Python packages

Python Package Installation


            $ pip3 install flask
            

          if __name__ == '__main__':
              print('hello world')
          
code hello.py

Storing Data, the right way

  • List - a mutable list of items
  • Dict - unsorted but a fast map of items
  • Set - like a dict without values
  • Tuple - the immutable and hashable list

List

Vector Like Implementation

Get Set Lenght Insert Remove
O(1) O(1) O(1) O(n) O(n)

List (cont’d)


            items = list(range(10))
            primes = [2, 3, 5, 7]

            print([item for item in items])
            print([item for item in items if item not in primes])
            print([item * 2 for item in items if item not in primes])
            
code list.py

Dict

__hash__(key)

Get Set Del
O(1) O(1) O(1)
Do you believe it ?

Set

Expression Explanation
spam & eggs
Every item in both
spam | eggs
Every item in either or both
spam - eggs
Every item in the first but not the latter
spam ^ eggs
Every item in either but not both
spam < eggs
True if every item in the first is contained in the latter

Tuple

There are few cases whre a tuple offers some really useful functionalities that a list does not.

Tuple (cont'd)


            t1 = 1, 3, 7, 3
            t2 = (1, 3, 7, 2)

            data = dict()
            data[t1] = 'me'
            data[t2] = 'who'

            import pprint

            pprint.pprint(data)
            
code tuple.py

Data Structure is fun

See Me

<Code Time.py>

Create Person class that can say hello and walk, then create Runner class that inherits from Person.


            class Person:
                def __init__(self, name, age):
                    self.name = name
                    self.age = age

                def say(self):
                    print('%s: Hello' % self.name)

                def walk(self):
                    print('Telgh Telgh')


            class Runner(Person):
                def __init__(self, name, age):
                    super().__init__(name, age)

                def walk(self):
                    print('Lekh Lekh')
            
code person.py

Lamba

Lambda expressions (sometimes called lambda forms) are used to create anonymous functions.


            lambda x, y: x * y + 1
            

So what ?!


            import functools

            foo = [1, 2, 3]
            functools.reduce(lambda x, y: x + y, foo)
            
code lambda.py

Scope in python, The LEGB Rule.

extension Local

Names assigned in any way within a function

extension Enclosing-function locals

Name in the local scope of any and all statically enclosing functions

Scope in python, The LEGB Rule. (cont’d)

extension Global

Names assigned at the top-level of a module file, or by executing a global statement in a def within the file

extension Built-in

Names preassigned in the built-in names module

Scope in python, The LEGB Rule. (cont’d)


            def f1():
                a = 1
                b = []

                def f2():
                    print(a)
                    b.append(1)
                print(b)
                f2()
                print(a)
                print(b)

            f1()
            
code scope.py

Python Packaging


            parent/
                __init__.py
                one/
                    __init__.py
                two/
                    __init__.py
                three/
                    __init__.py
            

Python Packaging (cont'd)

Python defines two types of packages, regular packages and namespace packages.

Regular packages are traditional packages as they existed in Python 3.2 and earlier.

Python Packaging (cont'd)

A regular package is typically implemented as a directory containing an __init__.py file. When a regular package is imported, this __init__.py file is implicitly executed, and the objects it defines are bound to names in the package’s namespace.

<Code Time.py>

Create calc package for four arithmetic operators (each operator has one file). Then create caculator application based on this package.

code calculator

Functions


            def function_name(param1, param2):
                '''
                I am your function documentation but you can use me
                in other and maybe bad ways.
                '''
                print('You saw me in the past')
            

Functions in Action


            function_name(1, 2)
            function_name(1, param2=2)
            function_name(param1=1, param2=2)
            

__doc__ is useful to provide some documentation

exec function supports dynamic execution of Python code

Function documentation, bad way


            def function_bad():
                '''print('Hello from the doc')'''
                pass
            print(exec(function_bad.__doc__))
            

I want a function with many many arguments

question_answer


            import functools


            def many_arg_func(list):
                s = functools.reduce(lambda x, y: x + y, list)
                return s

            many_arg_func([1, 2, 3, 4])
            

But I want to call my function normally :\

*args, **kwargs

The syntax is the * and **. The names *args and **kwargs are only by convention but there's no hard requirement to use them.

You would use *args when you're not sure how many arguments might be passed to your function

Similarly, **kwargs allows you to handle named arguments that you have not defined in advance


            def hello(param1, *args, **kwargs):
                print(param1, args, kwargs)

            hello(1, 2, 3, 4, 5, h=6)
            

            import functools

            def many_arg_func(*args):
                s = functools.reduce(lambda x, y: x + y, list)
                return s

            many_arg_func([1, 2, 3, 4])
            

Fibonacci

F(n) = F(n - 1) + F(n - 2)

<Code Time.py>

Write recuresive function for calculating Nth term in the Fibonacci sqeuence

0: 1, 1: 1, 2: 2, 3: 3, 4: 5, ...


            def fib(n):
                if n == 0 or n == 1:
                    return 1
                return fib(n - 1) + fib(n - 2)
            print(fib(10)
            
code fibonacci.py

Function call tree

Memoization

In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

<Code Time.py>

Write recuresive function with memoization for calculating Nth term in the Fibonacci sqeuence

0: 1, 1: 1, 2: 2, 3: 3, 4: 5, ...


            memory = {}

            def m_fibonacci(n):
                if n == 1 or n == 0:
                    return 1
                if n in memory:
                    return memory[n]
                memory[n] = m_fibonacci(n - 1) + m_fibonacci(n - 2)
                return memory[n]

            print(m_fibonacci(10))
            
code m-fibonacci.py

Functions are first-level citizens in python

To say that functions are first-class in a certain programming language means that they can be passed around and manipulated similarly to how you would pass around and manipulate other kinds of objects.


            def a_hello():
                print("Hi I am function A")


            a_hello()
            print(a_hello)
            b_hello = a_hello
            b_hello()
            1()
            
code function-citizen.py

no need to repeat all arguments every time


            def partial(f, *args):
                param = args

                def _f(*args, **kwargs):
                    return f(*param, *args, **kwargs)

                return _f


            def say(name, message):
                print(f'{message} to {name}')


            say('parham', 'hello')

            say_to_parham = partial(say, 'parham')
            say_to_parham('bye bye')
            
code partial.py

Event Handling

You can pass your callback into library for specific event and then customize its behaviour


            import paho.mqtt.client as mqtt


            # The callback for when the client receives a CONNACK response from the server.
            def on_connect(client, userdata, flags, rc):
                print("Connected with result code "+str(rc))

                # Subscribing in on_connect() means that if we lose the connection and
                # reconnect then subscriptions will be renewed.
                client.subscribe("$SYS/#")


              # The callback for when a PUBLISH message is received from the server.
              def on_message(client, userdata, msg):
                  print(msg.topic+" "+str(msg.payload))

              client = mqtt.Client()
              client.on_connect = on_connect
              client.on_message = on_message

              client.connect("iot.ceit.aut.ac.ir", 58904, 60)

              # Blocking call that processes network traffic, dispatches callbacks and
              # handles reconnecting.
              # Other loop*() functions are available that give a threaded interface and a
              # manual interface.
              client.loop_forever()
            

Decorators

New in Python 2.4


            def my_decorator(some_function):
                def wrapper():

                    print("Something is happening before some_function() is called.")

                    some_function()

                    print("Something is happening after some_function() is called.")

                return wrapper


            def just_some_function():
                print("Wheee!")


            just_some_function = my_decorator(just_some_function)
            just_some_function()
            

Decorators (cont'd)


            def my_decorator(some_function):
                def wrapper():

                    print("Something is happening before some_function() is called.")

                    some_function()

                    print("Something is happening after some_function() is called.")

                return wrapper


            @my_decorator
            def just_some_function():
                print("Wheee!")


            just_some_function()
            

Decorators (cont'd)

A Python decorator is a specific change to the Python syntax that allows us to more conveniently alter functions and methods.

A decorator is just another function which takes a function and returns one.

Function Wrapping


            import functools

            def my_decorator(some_function):
                @functools.wraps(some_function)
                def wrapper():

                    print("Something is happening before some_function() is called.")

                    some_function()

                    print("Something is happening after some_function() is called.")

                return wrapper


            @my_decorator
            def just_some_function():
                print("Wheee!")


            just_some_function()
            

Function Wrapping (cont'd)

Without the use of this decorator factory, the name of the just_some_function function would have been 'wrapper', and the docstring of the original example() would have been lost.

<Code Time.py>

Using flask create simple web server for handling /hello get request.


            import flask


            app = flask.Flask('Hello')


            @app.route('/hello')
            def hello():
                return '

Hello world

' app.run()
code hello-flask.py

Parameter in decorators


            @decorator
            def function_name():
                pass
            

decorator is a name for decorator function and will not actually call the function

Parameter in decorators (cont'd)

How to pass parameters into decorator when we are not caller

Parameter in decorators (cont'd)

Get parameters and create decorator in place.


            import functools


            def decorator_args(arg1, arg2):
                def decorator(f):
                    @functools.wraps(f)
                    def wrapper():
                        print(arg1)
                        f()
                        print(arg2)
                    return wrapper
                return decorator
            
code decorator-args.py

<Code Time.py>

Giving input from user and running appropriate functions for his/her entered commands.

Class


            class Person:
                '''
                Person class is used for representation person
                '''
                def say(self):
                    print("Hello")
            

Multi-Inheritance


            class SubClassName(ParentClass1[, ParentClass2]):
                """
                Optional class documentation string
                """
                # class code
            

C3 Linearization

The C3 superclass linearization is an algorithm used primarily to obtain the order in which methods should be inherited (the "linearization") in the presence of multiple inheritance, and is often termed Method Resolution Order (MRO).

@staticmethod

A static method does not receive an implicit first argument.

@classmethod

A class method receives the class as implicit first argument, just like an instance method receives the instance.

@classmethod


            class C:
                @classmethod
                def f(cls, arg1, arg2, ...): ...
            

@staticmethod


            class C:
                @staticmethod
                def f(arg1, arg2, ...): ..
            

<Code Time.py>

By using @classmethod create inner class database.

Duck Typing

It is a term used in dynamic languages that do not have strong typing.

The idea is that you don't need a type in order to invoke an existing method on an object - if a method is defined on it, you can invoke it.

If it looks like a duck and quacks like a duck, it's a duck

Boolean

__bool__

Called to implement truth value testing and the built-in operation bool()

Boolean (cont'd)


            class IsBool:
                def __init__(self, age):
                    self.age = age

                def __bool__(self):
                    return self.age > 20

            if is_bool:
                print("is_bool.age > 20")
            else:
                print("is_bool.age <= 20")
            
code bool.py

Wrap

Extend

Invent

@abc.abstractmethod

@abc.abstractclassmethod

@abc.abstractstaticmethod

<Code Time.py>

Create Person class with abstract walk method and say method, then create Runner class that inherits from Person.

Person Abstract Class


            import abc


            class Person(abc.ABC):
                def say(self):
                    print('Hello')

                @abc.abstractmethod
                def walk(self):
                    raise NotImplementedError()
            

ABC

This module provides the infrastructure for defining abstract base classes (ABCs) in Python

  • A way to overload isinstance() and issubclass().

class abc.ABC
An abstract base class can be created by simply deriving from ABC.

Meta-Class

meta-class

Meta-Class (cont'd)

Classes are callable. These objects normally act as factories for new instances of themselves, but variations are possible for class types that override __new__(). The arguments of the call are passed to __new__() and, in the typical case, to __init__() to initialize the new instance.

Meta-Class (cont'd)

Instances of arbitrary classes can be made callable by defining a __call__() method in their class.

Meta-Class (cont'd)

Once the appropriate metaclass has been identified, then the class namespace is prepared. If the metaclass has a __prepare__ attribute, it is called as namespace = metaclass.__prepare__(name, bases, **kwds) (where the additional keyword arguments, if any, come from the class definition).

Meta-Class (Python 3.0)


            class ParhamClass(type):

                def __new__(cls, name, bases, namespace, **kwargs):
                    print('cls: ', cls)
                    print('name: ', name)
                    print('bases: ', bases)
                    print('namespace: ', namespace)
                    print('kwargs: ', kwargs)
                    result = super().__new__(cls, name, bases, namespace)

                    return result


            class A(str, when='never', metaclass=ParhamClass):
                name = 'parham'

                def one(): pass

                def two(): pass

                def three(): pass

                def four(): pass
            
code meta-class-1.py

Meta-Class (Python 3.6)


            class BotFather:
                bots = []

                @classmethod
                def __init_subclass__(cls, name, **kwargs):
                    super().__init_subclass__(**kwargs)
                    print("New subclass from BotFather")
                    BotFather.bots.append(cls)

            class HelloBot(BotFather, name='hello'):
                def __init__(self):
                    print('New instance from HelloBot')
            
code meta-class-2.py

<Code Time.py>

Let's create plugin system, write parent class which holds a collection of loaded plugin.

Learn class customization in deep

See Me

What are descriptors?

Descriptors were introduced to Python way back in version 2.2.

They provide the developer with the ability to add managed attributes to objects.

The Descriptor Protocol

If any of these methods are defined for an object, it is said to be a descriptor.

  • __set__
  • __get__
  • __del__

__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). owner is always the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner. This method should return the (computed) attribute value or raise an AttributeError exception.

__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.

__del__(self, instnace)

Called to delete the attribute on an instance instance of the owner class.

__set_name__(self, owner, name)

New in version 3.6.

Called at the time the owning class owner is created. The descriptor has been assigned to name.


            class RevealAccess:
                '''
                A data descriptor that sets and returns values
                normally and prints a message logging their access.
                '''

                def __init__(self, initval=None, name='var'):
                    self.val = initval
                    self.name = name

                def __get__(self, instance, owner):
                    print('Retrieving', self.name, instance, owner)
                    return self.val

                def __set__(self, instance, val):
                    print('Updating', self.name, instance, val)
                    self.val = val

                def __set_name__(self, owner, name):
                    print('Set name', name, owner)
            
code descriptor.py

<Code Time.py>

Let's see SQLAlchemy Object-Relational Mapping (ORM)

Properties

Calling property() is a succinct way of building a data descriptor that triggers function calls upon access to an attribute.


            property(fget=None, fset=None, fdel=None, doc=None) -> property attribute
            

Properties in Action


            class Celsius:
                def __init__(self, temperature = 0):
                    self._temperature = temperature

                def to_fahrenheit(self):
                    return (self.temperature * 1.8) + 32

                @property
                def temperature(self):
                    print("Getting value")
                    return self._temperature

                @temperature.setter
                def temperature(self, value):
                    if value < -273:
                        raise ValueError("Temperature below -273 is not possible")
                    print("Setting value")
                    self._temperature = value
            
code properties.py

Kivy or Kiwi

Open source Python library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps.

Do you know JavaFX ?

Kivy Installation

Hello Kivy


            from kivy.app import App
            from kivy.uix.button import Button

            class TestApp(App):
                def build(self):
                    return Button(text='Hello World')

            TestApp().run()
            

KV Design Language

Kivy provides a design language specifically geared towards easy and scalable GUI Design. The language makes it simple to separate the interface design from the application logic, adhering to the separation of concerns principle.

KV Design Language (cont'd)

There are two ways to load Kv code into your application:

  • By name convention
  • 
                  MyApp -> my.kv
                  
  • Builder
  • 
                  Builder.load_file('path/to/file.kv')
                  

Events

  • Clock Events
  • Input Events
  • Class Events

Layouts

Layouts are containers used to arrange widgets in a particular manner.

  • Anchor Layout: Widgets can be anchored to the top, bottom, left, right or center.
  • Box Layout: Widgets are arranged sequentially, in either a vertical or a horizontal orientation.
  • GridLayout: Widgets are arranged in a grid defined by the rows and cols properties.

Threading

In computer science, a thread of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler, which is typically a part of the operating system.

Thread

The Thread class represents an activity that is run in a separate thread of control.

<Code Time.py>

Flask in single thread and multi-thread modes

Multi-Threading Flask


            app.run(threaded=True)
            

Threading in action


            import threading


            def run():
                print('some works')


            threading.Thread(target=run).start()
            
code thread.py

Thread.join(timeout=None)

Wait until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception – or until the optional timeout occurs.

Lock

A primitive lock is a synchronization primitive that is not owned by a particular thread when locked.

Locks also support the context management protocol.


            with lck:
                # some awesome synchronized works
            

Generators

Generators functions allow you to declare a function that behaves like an iterator, i.e. it can be used in a for loop.


            def firstn(n):
                sum = 0
                while num < n:
                    yield num
                    num += 1


            for i in firstn(10):
                print(i)
            
code generators.py

<Code Time.py>

Let's write fibonacci with generators.

Async-Await

The newer and cleaner syntax is to use the async/await keywords. Introduced in Python 3.5, async is used to declare a function as a coroutine. It can be applied to the function by putting it at the front of the definition.

Async-Await (cont'd)


            async def ping_server():
                # ping code here

            async def ping_local():
                return await ping_server('127.0.0.1')
            

Running the event loop

None of the coroutine stuff will matter (or work) if you don't know how to start and run an event loop.

The event loop is the central point of execution for asynchronous functions

Running the event loop (cont'd)


            loop = asyncio.get_event_loop()  
            loop.run_until_complete(speak_async())  
            loop.close()