Effective Python

第 6 章 內建模組

Speaker: 毛毛 (2017/03/04)

Outline

  • 42 以 functools.wrap 定義函式裝飾器
  • 43 考慮使用 contextlib 與 with 述句來建立可重用的 try/finally 行為
  • 44 用 copyreg 來使 pickle 更可靠
  • 45 本地時鐘使用 datetime 而非 time
  • 46 使用內建的演算法與資料結構
  • 47 精確度很重要時就使用 decimal
  • 48 知道去哪找社群建置的模組

42 以 functools.wrap 定義函式裝飾器

裝飾器 (decorator)

  • 套用在函式上
  • 能在被包裹的函式被呼叫的前後執行額外的程式碼
    • 可修改被包裹的函式的傳入值與回傳值
    • 除錯 (debug)
    • 註冊函式

直接來看個簡單的例子!

在任意函式的開頭與結尾印出指定訊息

In [11]:
# 裝飾器
def trace(func):
    def wrapper(*args, **kwargs):
        print("====== {} start ======".format(func.__name__))
        func(*args, **kwargs)
        print("======= {} end =======".format(func.__name__))
    return wrapper

def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

def sub(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1-num2))
    
add = trace(add)
add(5, 10)

sub = trace(sub)
sub(10, 5)
====== add start ======
5 + 10 = 15
======= add end =======
====== sub start ======
10 + 5 = 5
======= sub end =======

利用 @ 精簡寫法

In [13]:
# 裝飾器
def trace(func):
    def wrapper(*args, **kwargs):
        print("====== {} start ======".format(func.__name__))
        func(*args, **kwargs)
        print("======= {} end =======".format(func.__name__))
    return wrapper

@trace
def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

@trace
def sub(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1-num2))
    
add(5, 10)
sub(10, 5)
====== add start ======
5 + 10 = 15
======= add end =======
====== sub start ======
10 + 5 = 5
======= sub end =======

但前面的寫法卻造成了副作用

In [16]:
print(add.__name__)    # except 'add'
print(add)
print(sub.__name__)    # expect 'sub'
print(sub)
wrapper
<function trace.<locals>.wrapper at 0x7f8898723ea0>
wrapper
<function trace.<locals>.wrapper at 0x7f8898723e18>

解法: 利用 functools 內建模組的 wrap 函式

  • 協助撰寫裝飾器的一個'裝飾器'
  • 將內層函式重要的 metadata 都複製到外層函式
    • __module__, __name__, __qualname__, __doc__, __annotations__
In [17]:
from functools import wraps

# 裝飾器
def trace(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("====== {} start ======".format(func.__name__))
        func(*args, **kwargs)
        print("======= {} end =======".format(func.__name__))
    return wrapper

@trace
def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

print(add.__name__)
print(add)
add
<function add at 0x7f88987066a8>

Bonus 1: 有參數的裝飾器

In [394]:
trace_level = 2
def trace(level=1):                            # 接收參數
    def outter_wrapper(func):                  # 接收 func
        @wraps(func)
        def inner_wrapper(*args, **kwargs):
            if level == trace_level:
                print("====== {} start ======".format(func.__name__))
                func(*args, **kwargs)
                print("======= {} end =======".format(func.__name__))
            else:
                func(*args, **kwargs)
        return inner_wrapper
    return outter_wrapper

@trace(1)
def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

@trace(2)
def sub(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1-num2))
    
add(5, 10)            # add = trace(1)(add)
sub(10, 5)            # sub = trace(2)(add)
print(add.__name__)
print(sub.__name__)
5 + 10 = 15
====== sub start ======
10 + 5 = 5
======= sub end =======
add
sub

Bonus 2: 沒有參數的、用 class 實現的裝飾器

  • 有實現 __call__ 的 class 可以被當作函式來呼叫
In [42]:
from functools import update_wrapper

class Trace(object):
    def __init__(self, func):
        self.func = func
        update_wrapper(self, func)
    
    def __call__(self, *args, **kwargs):
        print("====== {} start ======".format(self.func.__name__))
        self.func(*args, **kwargs)
        print("======= {} end =======".format(self.func.__name__))

@Trace
def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

add(10, 20)            # add = Trace(add)
print(add.__name__)
====== add start ======
10 + 20 = 30
======= add end =======
add

Bonus 3: 有參數的、用 class 實現的裝飾器

In [395]:
trace_level = 2
class Trace(object):
    def __init__(self, level=1):
        self.level = level
    
    def __call__(self, func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            if self.level == trace_level:
                print("====== {} start ======".format(func.__name__))
                func(*args, **kwargs)
                print("======= {} end =======".format(func.__name__))
            else:
                func(*args, **kwargs)
        return wrapper

@Trace(1)
def add(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1+num2))

@Trace(2)
def sub(num1, num2):
    print("{} + {} = {}".format(num1, num2, num1-num2))
    
add(5, 10)            # add = Trace(1)(add)
sub(10, 5)            # sub = Trace(2)(sub)
print(add.__name__)
print(sub.__name__)
5 + 10 = 15
====== sub start ======
10 + 5 = 5
======= sub end =======
add
sub

Bonus 4: 疊加裝飾器

In [55]:
base_price = 100

def add_milk(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("====== Add milk =======")
        price = func(*args, **kwargs) + 30
        print("====== Done milk ======")
        return price
    return wrapper

def add_choco(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("====== Add choco =======")
        price = func(*args, **kwargs) + 10
        print("====== Done choco ======")
        return price
    return wrapper

def add_matcha(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("====== Add matcha ======")
        price = func(*args, **kwargs) + 20
        print("====== Done matcha =====")
        return price
    return wrapper
In [56]:
@add_milk
def coffee1():
    return base_price

print("coffee1 NT{}".format(coffee1()))
====== Add milk =======
====== Done milk ======
coffee1 NT130
In [57]:
@add_choco
@add_milk
def coffee2():
    return base_price

@add_milk
@add_choco
def coffee3():
    return base_price

print("coffee2 NT{}".format(coffee2()))        # coffee2 = add_choco(add_milk(coffee2))
print()
print("coffee3 NT{}".format(coffee3()))        # coffee3 = add_milk(add_choco(coffee3))
====== Add choco =======
====== Add milk =======
====== Done milk ======
====== Done choco ======
coffee2 NT140

====== Add milk =======
====== Add choco =======
====== Done choco ======
====== Done milk ======
coffee3 NT140
In [58]:
@add_matcha
@add_choco
@add_milk
def coffee4():
    return base_price

print("coffee4 NT{}".format(coffee4()))        # coffee4 = add_matcha(add_choco(add_milk(coffee4)))
====== Add matcha ======
====== Add choco =======
====== Add milk =======
====== Done milk ======
====== Done choco ======
====== Done matcha =====
coffee4 NT160

43 考慮使用 contextlib 與 with 述句來建立可重用的 try/finally 行為

with 述句

  • 可用於例外處理
  • 可用於工作的事前設置與事後清理
  • try/finally 的替代
  • 適用於對資源進行訪問的工作

我們來看個例子

上下兩種寫法的目的是一樣的

  • 不管是正常執行結束或是有例外發生,都要關閉檔案
In [284]:
try:
    f = open("/home/maomao/public_html/my_numbers.txt")
    print(f.read())
finally:
    f.close()
15
35
80

In [285]:
with open("/home/maomao/public_html/my_numbers.txt") as f:
    print(f.read())
15
35
80

讓自定義的物件能被用在 with 中

  • 實現 __enter____exit__
In [303]:
class Test(object):
    def __init__(self, num):
        print("初始化")
        self.my_list = [i for i in range(num)]
    
    def __enter__(self):
        print("前置工作")
        return self                                  # 賦值給 as 後面的變數
        
    def __exit__(self, e_type, e_value, e_trace):    # 捕捉 error
        print("清理工作")
        print(e_type)
        print(e_value)
        print(e_trace)

with Test(5) as obj:
    print(obj.my_list)
初始化
前置工作
[0, 1, 2, 3, 4]
清理工作
None
None
None
In [304]:
with Test(5) as obj:
    print(obj.my_list)
    print(obj.my_list2)
初始化
前置工作
[0, 1, 2, 3, 4]
清理工作
<class 'AttributeError'>
'Test' object has no attribute 'my_list2'
<traceback object at 0x7f88985f93c8>
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-304-1f9643b11a2c> in <module>()
      1 with Test(5) as obj:
      2     print(obj.my_list)
----> 3     print(obj.my_list2)

AttributeError: 'Test' object has no attribute 'my_list2'

例外處理

In [343]:
class Test(object):
    def __init__(self, num):
        print("初始化")
        self.my_list = [i for i in range(num)]
    
    def __enter__(self):
        print("前置工作")
        return self
        
    def __exit__(self, e_type, e_value, e_trace):
        print("清理工作")
        print(e_type)
        print(e_value)
        print(e_trace)
        if e_type == AttributeError:
            return True

with Test(5) as obj:
    print(obj.my_list)
    print(obj.my_list2)
初始化
前置工作
[0, 1, 2, 3, 4]
清理工作
<class 'AttributeError'>
'Test' object has no attribute 'my_list2'
<traceback object at 0x7f88887189c8>
In [377]:
import traceback

class Test(object):
    def __init__(self, num):
        print("初始化")
        self.my_list = [i for i in range(num)]
    
    def __enter__(self):
        print("前置工作")
        return self
        
    def __exit__(self, e_type, e_value, e_trace):
        print("清理工作")
        if e_type == AttributeError:
            traceback.print_exception(e_type, e_value, e_trace)
            return True

with Test(5) as obj:
    print(obj.my_list)
    print(obj.my_list2)
初始化
前置工作
[0, 1, 2, 3, 4]
清理工作
Traceback (most recent call last):
  File "<ipython-input-377-1c57f3d7e706>", line 20, in <module>
    print(obj.my_list2)
AttributeError: 'Test' object has no attribute 'my_list2'

注意!

在進入 __enter__ 前就發生例外,__exit__ 是管不到的

  • with 述句根本還沒發生效用
In [289]:
with Test("a") as obj:
    print(obj.my_list)
初始化
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-289-bb2b0635a0c8> in <module>()
----> 1 with Test("a") as obj:
      2     print(obj.my_list)

<ipython-input-288-fd7be543709b> in __init__(self, num)
      2     def __init__(self, num):
      3         print("初始化")
----> 4         self.my_list = [i for i in range(num)]
      5 
      6     def __enter__(self):

TypeError: 'str' object cannot be interpreted as an integer

contextmanager 裝飾器

  • 讓自定義的物件和函式能用在 with 述句中
  • 比藉由 __enter____exit__ 定義一個新類別 (標準作法) 容易多了

我們來看一個新例子!

logging

  • with 述句之外的 logging level 是 error
  • with 述句之內的 logging level 可動態調整
In [387]:
import logging

logger = logging.getLogger()
logger.setLevel(logging.ERROR)

def my_log(num):
    logging.info("Info msg {}".format(num))
    logging.debug("Debug msg {}".format(num))
    logging.warning("Warning msg {}".format(num))
    logging.error("Error msg {}".format(num))

my_log(1)
ERROR:root:Error msg 1
In [388]:
from contextlib import contextmanager

@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield
    finally:
        logger.setLevel(old_level)
    
with debug_logging(logging.WARNING):
    my_log(1)

my_log(2)
WARNING:root:Warning msg 1
ERROR:root:Error msg 1
ERROR:root:Error msg 2

解析

  • yield 之前: 等同於 __enter__
  • yield: 賦值給 as 後面的變數
  • yield 之後: 等同於 __exit__

例外處理

In [389]:
# 未處理例外

@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield
    finally:
        logger.setLevel(old_level)
    
with debug_logging(logging.WARNING):
    my_log(123, 123)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-389-5f344047209d> in <module>()
     12 
     13 with debug_logging(logging.WARNING):
---> 14     my_log(123, 123)

TypeError: my_log() takes 1 positional argument but 2 were given
In [397]:
# 有處理例外 1

@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield
    finally:
        logger.setLevel(old_level)
        return True
    
with debug_logging(logging.WARNING):
    my_log(123, 123)
In [398]:
# 有處理例外 2

@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield
    except TypeError as e:
        print(e)
    finally:
        logger.setLevel(old_level)
    
with debug_logging(logging.WARNING):
    my_log(123, 123)
my_log() takes 1 positional argument but 2 were given

賦值給 as 後面的變數

In [393]:
@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield logger
    finally:
        logger.setLevel(old_level)
    
with debug_logging(logging.WARNING) as logger:
    logger.info("Info log 1")
    logger.debug("Debug log 1")
    logger.warning("Warning log 1")
    logger.error("Error log 1")
    
with debug_logging(logging.INFO) as logger:
    logger.info("Info log 2")
    logger.debug("Debug log 2")
    logger.warning("Warning log 2")
    logger.error("Error log 2")
    
logging.info("Info log 3")
logging.debug("Debug log 3")
logging.warning("Warning log 3")
logging.error("Error log 3")
WARNING:root:Warning log 1
ERROR:root:Error log 1
INFO:root:Info log 2
WARNING:root:Warning log 2
ERROR:root:Error log 2
ERROR:root:Error log 3

44 用 copyreg 來使 pickle 更可靠

pickle 內建模組

  • 將 Python 物件序列化 (serialize) 為位元串流 (stream of bytes)
  • 將位元組資料解序列化 (deserialize) 為 Python 物件

直接來看個例子吧!

保存遊戲狀態

In [399]:
class GameState(object):
    def __init__(self):
        self.level = 0
        self.lives = 4

state = GameState()
state.level += 1
state.lives -= 1
print(state.__dict__)
print(state)
{'lives': 3, 'level': 1}
<__main__.GameState object at 0x7f887a744358>
In [400]:
import pickle

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)

with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
print(state_record)
b'\x80\x03c__main__\nGameState\nq\x00)\x81q\x01}q\x02(X\x05\x00\x00\x00livesq\x03K\x03X\x05\x00\x00\x00levelq\x04K\x01ub.'
{'lives': 3, 'level': 1}
<__main__.GameState object at 0x7f887a7442b0>
In [169]:
game_state = pickle.dumps(state)
print(game_state)
state_record = pickle.loads(game_state)
print(state_record.__dict__)
print(state_record)
b'\x80\x03c__main__\nGameState\nq\x00)\x81q\x01}q\x02(X\x05\x00\x00\x00livesq\x03K\x03X\x05\x00\x00\x00levelq\x04K\x01ub.'
{'lives': 3, 'level': 1}
<__main__.GameState object at 0x7f88985d7c88>

注意 !

如果物件的參照不在的話,會無法解序列化

In [170]:
del GameState
state_record = pickle.loads(game_state)
print(state_record.__dict__)
print(type(state_record))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-170-24e482050697> in <module>()
      1 del GameState
----> 2 state_record = pickle.loads(game_state)
      3 print(state_record.__dict__)
      4 print(type(state_record))

AttributeError: Can't get attribute 'GameState' on <module '__main__'>

新版本遊戲+舊版本遊戲狀態+backwards-compatible

In [415]:
# old version
class GameState(object):
    def __init__(self):
        self.level = 0
        self.lives = 4

state = GameState()
state.level += 1
state.lives -= 1

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)
    
with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
b'\x80\x03c__main__\nGameState\nq\x00)\x81q\x01}q\x02(X\x05\x00\x00\x00livesq\x03K\x03X\x05\x00\x00\x00levelq\x04K\x01ub.'
{'lives': 3, 'level': 1}
In [416]:
# new version
class GameState(object):
    def __init__(self):
        self.level = 0    # still exist
        self.lives = 4    # still exist
        self.point = 0    # new

with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)    # no point
print(state_record)
{'lives': 3, 'level': 1}
<__main__.GameState object at 0x7f88886646a0>

copyreg 內建模組

  • 註冊負責序列化 Python 物件的函式
  • 控制 pickle 的行為,讓它變得更可靠
In [405]:
import copyreg

# old version 1
class GameState(object):
    def __init__(self):
        self.level = 0
        self.lives = 4

def unpickle_game_state(kwargs):                 #
    obj = GameState()
    obj.__dict__.update(kwargs)
    return obj

def pickle_game_state(game_state):               #
    kwargs = game_state.__dict__
    return unpickle_game_state, (kwargs,)

copyreg.pickle(GameState, pickle_game_state)     #

state = GameState()
state.level += 1
state.lives -= 1

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)
    
with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
b'\x80\x03c__main__\nunpickle_game_state\nq\x00}q\x01(X\x05\x00\x00\x00livesq\x02K\x03X\x05\x00\x00\x00levelq\x03K\x01u\x85q\x04Rq\x05.'
{'lives': 3, 'level': 1}
In [406]:
# new version 1
class GameState(object):
    def __init__(self):
        self.level = 0
        self.lives = 4
        self.point = 0

with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
{'lives': 3, 'point': 0, 'level': 1}
In [413]:
# old version 2
class GameState(object):
    def __init__(self, level=0, lives=4):        #
        self.level = level
        self.lives = lives

def unpickle_game_state(kwargs):                 #
    return GameState(**kwargs)

def pickle_game_state(game_state):               #
    kwargs = game_state.__dict__
    print(game_state)
    return unpickle_game_state, (kwargs,)

copyreg.pickle(GameState, pickle_game_state)     #

state = GameState()
state.level += 1
state.lives -= 1

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)
    
with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
<__main__.GameState object at 0x7f8888664710>
b'\x80\x03c__main__\nunpickle_game_state\nq\x00}q\x01(X\x05\x00\x00\x00livesq\x02K\x03X\x05\x00\x00\x00levelq\x03K\x01u\x85q\x04Rq\x05.'
{'lives': 3, 'level': 1}
In [414]:
# new version 2
class GameState(object):
    def __init__(self, level=0, lives=4, point=0):
        self.level = level
        self.lives = lives
        self.point = point

with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
{'lives': 3, 'point': 0, 'level': 1}

新版本遊戲+舊版本遊戲狀態+backwards-incompatible

In [429]:
# old version 2
class GameState(object):
    def __init__(self, level=0, lives=4):        #
        self.level = level
        self.lives = lives

def unpickle_game_state(kwargs):                 #
    return GameState(**kwargs)

def pickle_game_state(game_state):               #
    kwargs = game_state.__dict__
    print(game_state)
    return unpickle_game_state, (kwargs,)

copyreg.pickle(GameState, pickle_game_state)     #

state = GameState()
state.level += 1
state.lives -= 1

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)
    
with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
<__main__.GameState object at 0x7f888866b470>
b'\x80\x03c__main__\nunpickle_game_state\nq\x00}q\x01(X\x05\x00\x00\x00livesq\x02K\x03X\x05\x00\x00\x00levelq\x03K\x01u\x85q\x04Rq\x05.'
{'lives': 3, 'level': 1}
In [430]:
class GameState(object):
    def __init__(self, lives=4):
        self.lives = lives

with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-430-9bb8871e64d2> in <module>()
      4 
      5 with open("/home/maomao/game_state.bin", "rb") as f:
----> 6     state_record = pickle.load(f)
      7 
      8 print(state_record.__dict__)

<ipython-input-429-9d82c2ebeb58> in unpickle_game_state(kwargs)
      6 
      7 def unpickle_game_state(kwargs):                 #
----> 8     return GameState(**kwargs)
      9 
     10 def pickle_game_state(game_state):               #

TypeError: __init__() got an unexpected keyword argument 'level'

版本控制

In [431]:
def unpickle_game_state(kwargs):
    version = kwargs.pop("version", 1)
    if version == 1:
        kwargs.pop("level")
    return GameState(**kwargs)

def pickle_game_state(game_state): 
    kwargs = game_state.__dict__
    kwargs["version"] = 2
    return unpickle_game_state, (kwargs,)

copyreg.pickle(GameState, pickle_game_state)

with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
{'lives': 3}

新版本遊戲 (名稱改變 or 路徑改變)+舊版本遊戲狀態

In [194]:
# old version
class GameState(object):
    def __init__(self, level=0, lives=4):        #
        self.level = level
        self.lives = lives

def unpickle_game_state(kwargs):                 #
    return GameState(**kwargs)

def pickle_game_state(game_state):               #
    kwargs = game_state.__dict__
    return unpickle_game_state, (kwargs,)

copyreg.pickle(GameState, pickle_game_state)     #

state = GameState()

with open("/home/maomao/game_state.bin", "wb") as f:
    pickle.dump(state, f)
    
with open("/home/maomao/game_state.bin", "rb") as f:
    print(f.read())
    
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
b'\x80\x03c__main__\nunpickle_game_state\nq\x00}q\x01(X\x05\x00\x00\x00livesq\x02K\x04X\x05\x00\x00\x00levelq\x03K\x00u\x85q\x04Rq\x05.'
{'lives': 4, 'level': 0}
In [195]:
del GameState

# new version
class GameState2(object):
    def __init__(self, level=0, lives=4):
        self.level = level
        self.lives = lives

def unpickle_game_state(kwargs):
    return GameState2(**kwargs)

copyreg.pickle(GameState2, pickle_game_state)
        
with open("/home/maomao/game_state.bin", "rb") as f:
    state_record = pickle.load(f)
    
print(state_record.__dict__)
{'lives': 4, 'level': 0}

注意 !!!

pickle 模組的序列化格式在設計上是不安全的

相較之下,json 模組的序列化格式在設計上比較安全

  • 序列化後的資料簡單描述了一個物件階層架構

45 本地時鐘使用 datetime 而非 time

Python 提供兩種方式來進行時區的轉換

  • time 內建模組
  • datetime 內建模組 + pytz 模組 (社群所建置的)

time 內建模組

  • platform-dependent
    • 不一定擁有各種時區的定義
    • 不一定能處理各種時區的時間轉換

將時間字串轉成時間物件,再將時間物件轉回時間字串

In [103]:
from time import strptime, strftime

time_format = "%Y-%m-%d %H:%M:%S"
parse_format = "%Y-%m-%d %H:%M:%S %Z"
time_str = "2017-03-02 15:45:16 GMT"
time_tuple = strptime(time_str, parse_format)
time_str = strftime(time_format, time_tuple)
print(time_str)
2017-03-02 15:45:16
In [104]:
time_format = "%Y-%m-%d %H:%M:%S"
parse_format = "%Y-%m-%d %H:%M:%S %Z"
time_str = "2017-03-02 15:45:16 EDT"
time_tuple = strptime(time_str, parse_format)
time_str = strftime(time_format, time_tuple)
print(time_str)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-104-e69ff47fe49c> in <module>()
      1 parse_format = "%Y-%m-%d %H:%M:%S %Z"
      2 time_str = "2017-03-02 15:45:16 EDT"
----> 3 time_tuple = strptime(time_str, parse_format)
      4 time_str = strftime(time_format, time_tuple)
      5 print(time_str)

/usr/lib/python3.4/_strptime.py in _strptime_time(data_string, format)
    492     """Return a time struct based on the input string and the
    493     format string."""
--> 494     tt = _strptime(data_string, format)[0]
    495     return time.struct_time(tt[:time._STRUCT_TM_ITEMS])
    496 

/usr/lib/python3.4/_strptime.py in _strptime(data_string, format)
    335     if not found:
    336         raise ValueError("time data %r does not match format %r" %
--> 337                          (data_string, format))
    338     if len(data_string) != found.end():
    339         raise ValueError("unconverted data remains: %s" %

ValueError: time data '2017-03-02 15:45:16 EDT' does not match format '%Y-%m-%d %H:%M:%S %Z'

pytz 模組

  • 提供你可能需要的各種時區的定義

將本地時間轉為其他時區的時間

In [140]:
from datetime import datetime
import pytz

# step1: 取得本地時間
now = "2017-03-02 20:40:00"
time_format = "%Y-%m-%d %H:%M:%S"
native_time = datetime.strptime(now, time_format)
taiwan = pytz.timezone("Asia/Taipei")
taiwan_now = taiwan.localize(native_time)
print(taiwan_now.isoformat())

# step2: 將本地時間轉為 UTC
utc_now = taiwan_now.astimezone(pytz.utc)
print(utc_now.isoformat())

# step3: 從 UTC 轉為其他時區
pacific = pytz.timezone("US/Pacific")
pacific_now = utc_now.astimezone(pacific)
pacific_now = pacific.normalize(pacific_now)
print(pacific_now.isoformat())
2017-03-02T20:40:00+08:00
2017-03-02T12:40:00+00:00
2017-03-02T04:40:00-08:00

normalize()

  • 可處理 unexisting time for DST (Daylight Saving Time)
In [141]:
# 2016-03-13 01:59:59 -> 2016-03-13 03:00:00

now = "2016-03-13 02:10:00"
time_format = "%Y-%m-%d %H:%M:%S"
native_time = datetime.strptime(now, time_format)
eastern = pytz.timezone("US/Eastern")
eastern_now = eastern.localize(native_time)
print(eastern_now.isoformat())
eastern_now = eastern.normalize(eastern_now)
print(eastern_now.isoformat())
2016-03-13T02:10:00-05:00
2016-03-13T03:10:00-04:00

localize(is_dst=xxx)

  • 可處理 ambiguous time for DST
  • xxx: True, False, None
In [142]:
# 2016-11-06 01:59:59 -> 2016-11-06 01:00:00

now = "2016-11-06 01:10:00"
time_format = "%Y-%m-%d %H:%M:%S"
native_time = datetime.strptime(now, time_format)
eastern = pytz.timezone("US/Eastern")
eastern_now = eastern.localize(native_time)
print(eastern_now.isoformat())
utc_now = eastern_now.astimezone(pytz.utc)
print(utc_now.isoformat())

print()

now = "2016-11-06 01:10:00"
time_format = "%Y-%m-%d %H:%M:%S"
native_time = datetime.strptime(now, time_format)
eastern = pytz.timezone("US/Eastern")
eastern_now = eastern.localize(native_time, is_dst=True)
print(eastern_now.isoformat())
utc_now = eastern_now.astimezone(pytz.utc)
print(utc_now.isoformat())
2016-11-06T01:10:00-05:00
2016-11-06T06:10:00+00:00

2016-11-06T01:10:00-04:00
2016-11-06T05:10:00+00:00

Summary

  • 搭配 pytz 來使用 datetime 內建模組,以便可靠地在不同時區的時間之間進行轉換
  • 處理過程中,永遠都以 UTC 來表示時間,在要呈現給使用者時才轉換成當地時間

46 使用內建的演算法與資料結構

Double-ended Queue

In [219]:
from collections import deque

fifo = deque()
fifo.append(1)
fifo.append(2)
print(fifo)
x = fifo.pop()
print(fifo)
fifo.appendleft(3)
print(fifo)
y = fifo.popleft()
print(fifo)
print(x, y)
deque([1, 2])
deque([1])
deque([3, 1])
deque([1])
2 3

Ordered Dictionary

In [237]:
# 一般的 dict 是無序的

a = {}
a["B"] = 2
a["A"] = 3
a["C"] = 1
print(a)
for i in a:
    print(i)
b = {}
b["C"] = 1
b["B"] = 2
b["A"] = 3
print(b)
for i in b:
    print(i)
{'A': 3, 'B': 2, 'C': 1}
A
B
C
{'A': 3, 'B': 2, 'C': 1}
A
B
C
In [238]:
from collections import OrderedDict

a = OrderedDict()
a["B"] = 2
a["A"] = 3
a["C"] = 1
print(a)
for i in a:
    print(i)
b = OrderedDict()
b["C"] = 1
b["B"] = 2
b["A"] = 3
print(b)
for i in b:
    print(i)
OrderedDict([('B', 2), ('A', 3), ('C', 1)])
B
A
C
OrderedDict([('C', 1), ('B', 2), ('A', 3)])
C
B
A
In [243]:
from collections import defaultdict

# int 內建函式會回傳 0
print(int())

stats = defaultdict(int)
print(stats.items())
stats["my_counter"] += 1
print(stats.items())
0
dict_items([])
dict_items([('my_counter', 1)])
In [240]:
stats = defaultdict(int, {"A": 123})
print(stats.items())
stats["B"]
print(stats.items())
dict_items([('A', 123)])
dict_items([('B', 0), ('A', 123)])

Heap Queue

  • 用來維護 priority queue 的實用資料結構
In [254]:
from heapq import heappush, heappop

a = []
heappush(a, 5)
heappush(a, 3)
heappush(a, 7)
heappush(a, 4)
print(a)                # 依照 priority 自動排序
print()
print(heappop(a))       # 移除項目時一定從 priority 最低的項目開始移除
print(a)
print(heappop(a))
print(a)
print(heappop(a))
print(a)
print(heappop(a))
print(a)
[3, 4, 7, 5]

3
[4, 5, 7]
4
[5, 7]
5
[7]
7
[]

nsmallest(n, iterable, key=None)

  • Find the n smallest elements in a dataset.
  • Equivalent to: sorted(iterable, key=key)[:n]
In [433]:
from heapq import nsmallest

a = []
heappush(a, 5)
heappush(a, 3)
heappush(a, 7)
heappush(a, 4)
assert a[0] == nsmallest(1, a)[0] == 3
assert a[:2] == nsmallest(2, a) == [3, 4]

Bisection

  • 提供二元搜尋 (binary search) 一個有序序列的有效方式
In [267]:
from time import time
from bisect import bisect_left

x = list(range(10**6))
t1 = time()
i = x.index(991234)
t2 = time()
i = bisect_left(x, 991234)
t3 = time()

index_cost = t2-t1
bisect_cost = t3-t2
print("list.index cost {}".format(index_cost))
print("bisect_left cost {}".format(bisect_cost))
if bisect_cost < index_cost:
    print("bisect_left win !!!")
elif index_cost < bisect_cost:
    print("list.index win !!!")
else:
    print("the two drew")
list.index cost 0.01529073715209961
bisect_left cost 6.604194641113281e-05
bisect_left win !!!

Iterator Tools

  • itertools 內建模組含有為數眾多的函式,可用來組織迭代器 (iterators) 並與之互動
    • 函式大致可分為三類: 將迭代器連結、過濾迭代器項目、結合源自迭代器的項目

Linking iterators together

  • chain: Combines multiple iterators into a single sequential iterator.
  • cycle: Repeats an iterator’s items forever.
  • tee: Splits a single iterator into multiple parallel iterators.
  • zip_longest: A variant of the zip built-in function that works well with iterators of different lengths
In [270]:
from itertools import chain

def iterator1():
    string = "maomao"
    for s in string:
        yield s

def iterator2():
    for i in range(5):
        yield i

for item in chain(iterator1(), iterator2()):
    print(item)
m
a
o
m
a
o
0
1
2
3
4
In [274]:
from itertools import cycle

def iterator3():
    for i in ["A", "B", "C"]:
        yield i

count = 0
for item in cycle(iterator3()):
    if count == 7:
        break
    print(item)
    count += 1
A
B
C
A
B
C
A

Filtering items from an iterator

  • islice: Slices an iterator by numerical indexes without copying.
  • takewhile: Returns items from an iterator while a predicate function returns True.
  • dropwhile: Returns items from an iterator once the predicate function returns False for the first time.
  • filterfalse: Returns all items from an iterator where a predicate function returns False. The opposite of the filter built-in function.
In [277]:
from itertools import islice

def iterator4():
    for i in range(10):
        yield i

for item in islice(iterator4(), None, None, 2):
    print(item)

print()
    
for item in islice(iterator4(), 4, 7):
    print(item)
0
2
4
6
8

4
5
6
In [278]:
from itertools import takewhile

def is_even(num):
    return num % 2 == 0

def iterator5():
    for i in [10, 12, 4, 6, 8, 11, 2, 9]:
        yield i

for item in takewhile(is_even, iterator5()):
    print(item)
10
12
4
6
8

Combinations of items from iterators

  • product: Returns the Cartesian product of items from an iterator, which is a nice alternative to deeply nested list comprehensions.
  • permutations: Returns ordered permutations of length N with items from an iterator.
  • combinations: Returns the unordered combinations of length N with unrepeated items from an iterator
In [281]:
from itertools import permutations

def iterator6():
    for i in "ABCD":
        yield i

for item in permutations(iterator6(), 2):
    print(item)
('A', 'B')
('A', 'C')
('A', 'D')
('B', 'A')
('B', 'C')
('B', 'D')
('C', 'A')
('C', 'B')
('C', 'D')
('D', 'A')
('D', 'B')
('D', 'C')
In [283]:
from itertools import combinations

def iterator7():
    for i in "ABCD":
        yield i

for item in combinations(iterator7(), 2):
    print(item)
('A', 'B')
('A', 'C')
('A', 'D')
('B', 'C')
('B', 'D')
('C', 'D')

47 精確度很重要時就使用 decimal

直接來看個例子吧

一通國際電話的費用

  • 電話費率: 1.45 美元/分
  • 通話時間: 3 分鐘 42 秒
In [ ]:
# Python2

rate = 1.45
seconds = 3*60 + 42
cost = rate * seconds / 60    # 5.364999999999999
print(cost)                   # 5.365
print(round(cost, 2))         # 5.36
In [72]:
# Python3

rate = 1.45
seconds = 3*60 + 42
cost = rate * seconds / 60    # 5.364999999999999
print(cost)
print(round(cost, 2))
5.364999999999999
5.36

另一通國際電話的費用

  • 電話費率: 0.05 美元/分
  • 通話時間: 5 秒
In [ ]:
# Python2

rate = 0.05
seconds = 5
cost = rate * seconds / 60    # 0.004166666666666667
print(cost)                   # 0.00416666666667
print(round(cost, 2))         # 0.0
In [65]:
# Python3

rate = 0.05
seconds = 5
cost = rate * seconds / 60    # 0.004166666666666667
print(cost)
print(round(cost, 2))
0.004166666666666667
0.0

預期費用的算法是 round up,但

  • Python2: 1-4 round down, 5-9 round up
  • Python3: round-towards-even

round-towards-even

In [68]:
# Python3

print(round(1.2))
print(round(1.5))    #
print(round(1.7))
print()
print(round(2.2))
print(round(2.5))    #
print(round(2.7))
1
2
2

2
2
3

decimal 內建模組的 Decimal 類別

  • 預設提供小數點後 28 位數的定點數學運算 (fixed point math)
  • 對 rounding 的行為有更多的控制權
In [74]:
from decimal import Decimal, ROUND_UP

rate = Decimal('1.45')
seconds = Decimal('222')                 # 3*60 + 42
cost = rate * seconds / Decimal('60')    # Decimal('5.365')
print(cost)
print(cost.quantize(Decimal('0.01'), rounding=ROUND_UP))
5.365
5.37
In [75]:
from decimal import Decimal, ROUND_UP

rate = Decimal('0.05')
seconds = Decimal('5')
cost = rate * seconds / Decimal('60')    # Decimal('0.004166666666666666666666666667')
print(cost)
print(cost.quantize(Decimal('0.01'), rounding=ROUND_UP))
0.004166666666666666666666666667
0.01

注意 !

Decimal 對於定點數字 (fixed point numbers) 而言運作良好,但精確度還是有限制

  • 要表示精確度沒有限制的有理數,可考慮使用 fractions 內建模組的 Fraction 類別

48 知道去哪找社群建置的模組

PyPI - the Python Package Index

  • Python 模組的中央儲藏庫
  • 由 Python 社群建置及維護
  • 大多數的 PyPI 模組都是自由且開源的
  • https://pypi.python.org/pypi

pip

  • 用來從 PyPI 安裝套件的命令列工具
  • Python 3.4 之後的版本預設都會安裝