High-level ML frameworks and libraries (e.g., PyTorch, JAX, TensorFlow, NumPy, Triton, and many more) are mostly based on Python.

I’ve known Python for a while, but I’ve never learned it to a professional degree and wouldn’t say I’m good at Python programming. So, I decided to read Fluent Python (which seems to be one of the ‘bible’ figures of Python) to cover some topics and improve my Python programming skills.

One thing that was hard for me while reading this book was that it was too dense to just understand and skip. I needed to type in the code, but even that wasn’t enough. I figured this is a dictionary-like book that is hard to read repeatedly, so I decided to write a much more compact, digestable, and straightforward course note that works as a cheat sheet but also keeps the context as much as possible.

Note: This is not designed to be a replacement of the book, but for a handy review note or cheat sheet used daily.

Let’s start.

Part 2. Data Structures

Chapter 2. Sequences

A sequence in Python is a general term for an ordered collection of items. This means the items have a specific order, and you can access them by their position (their index).

We can divide sequences into types that can hold items of different type(Container sequence) or types that can’t (Flat sequence). Flat sequences are more compact but are limited to holding primitive values.

Another way to divide is by mutability(Mutable sequences vs Immutable sequences).

Listcomps and Genexps

List Comprehension, or Listcomp
this creates list with single line without using for loop or .append()

>>> arr = [1,2,3]  
>>> ex = [-x for x in arr]  
>>> ex  
[-1, -2, -3]

listcomp is even faster than map or filter at least in some cases.
(Try running this in Python Shell if you are curious!)

Cartesian Product
this is just nested listcomp (double for loop).

>>> colors = ['black', 'white']
>>> sizes = ['S', 'M', 'L']
>>> tshirts = [(color, size) for color in colors for size in sizes]  
>>> tshirts
[('black', 'S'), ('black', 'M'), ('black', 'L'), ('white', 'S'),
('white', 'M'), ('white', 'L')]

Generator Expression, or Genexps

what this is/does: similar to listcomps but uses parantheses instead of square brackets. This saves memory because it yields items one by one using the iterator protocol instead of building a whole list and proceeding.

listcomp is expensive but reusable, generator is cheap but cannot be used again.

Tuples

Tuples as Records
Since tuples are immutable, they (to be more accurate, their location) can be used as records.

Tuple Unpacking
we can unpack tuples that have two or more values by assigning items from an iterable to a tuple of variables.

This is an example of tuple used as a record and unpacking it.

>>> names = [('junu', 'park'), ('john', 'doe')]  
>>> for first_name, _ in names:  
... print(first_name)  
...  
junu  
john

we can use *args while unpacking tuple to extract the part(s) of what we want.

>>> names = [('junu', 'park'), ('john', 'doe'), ('ruby', 'onRails'), ('elon', 'musk'), ('the', 'primagen')]
>>> a, b, *c = names
>>> a
('junu', 'park')
>>> b
('john', 'doe')
>>> c
[('ruby', 'onRails'), ('elon', 'musk'), ('the', 'primagen')]

note that we can only use one args each time.

Nested Tuple Unpacking

If the expression of nested tuples match, Python will match the nesting structure and unpack it properly.

this is a non-realistic example but you will understand how nested tuple works by this example:

>>> nested_tuple = [('a', ('b', ('c', ('d')))), ('e', ('f', ('g', ('h'))))]
>>> for (x, (y, (z, (w)))) in nested_tuple:
...     print(x, y, z, w)
...
a b c d
e f g h

Named Tuples
This enhances the names(field names and a class name) of tuple.

>>> from collections import namedtuple
>>> Name = namedtuple('Name', 'first_name last_name')
>>> junu = Name('Junu', 'Park')
>>> elon = Name('Elon', 'Musk')
>>> prime = Name('The', 'Primagen')
>>> junu
Name(first_name='Junu', last_name='Park')
>>> elon
Name(first_name='Elon', last_name='Musk')
>>> prime
Name(first_name='The', last_name='Primagen')

As you can see there are two parameters for namedtuple: first one is the class name and second are field names seperated by single space.

We can use additional methods supported in namedtuples such as,

Name(first_name='The', last_name='Primagen')
>>> prime.last_name
'Primagen'
>>> prime._asdict()
{'first_name': 'The', 'last_name': 'Primagen'}

Slicing

You can slice not only lists but other sequence types like strings or tuples.

Why slices and range exclude last item
because it works well with the zero-based(starting from 0) indexing used in Python.

More specifically, it’s

• easy to see the length of a slice or range when only the stop position is given.
• easy to compute the length of a slice or range when start and stop are given by subtracting them.
• easy to split a sequence in two parts at any index x using the same x without overlapping.

Using $+$ and $*$ with Sequences

Multiplying list with integer copies of the same sequence works.

>>> l = [1,2,3]  
>>> l2 = l * 3  
>>> l2  
[1, 2, 3, 1, 2, 3, 1, 2, 3]  
>>> id(l)  
139738068947456  
>>> id(l2)  
139738067406976

Building Lists of Lists

The problem appears when the elements in the list are mutable items. The most common case is building lists of lists.

>>> board1 = [['_'] * 3 for i in range(3)]  
>>> board1  
[['_', '_', '_'], ['_', '_', '_'], ['_', '_', '_']]  
>>> board1[1][2] = 'x'  
>>> board1  
[['_', '_', '_'], ['_', '_', 'x'], ['_', '_', '_']]
>>> id(board1[0])
139738042877824
>>> id(board1[1])
139738043505344
>>> id(board1[2])
139738043329344

board1 works fine, meaning it has different address for each sub arrays.

but if we multiply list, it just references the same list:

>>> board2 = [['_'] * 3] * 3
>>> board2
[['_', '_', '_'], ['_', '_', '_'], ['_', '_', '_']]
>>> board[1][2] = 'x'
>>> board2
[['_', '_', '_'], ['_', '_', '_'], ['_', '_', '_']]
>>> board2[1][2] = 'x'
>>> board2
[['_', '_', 'x'], ['_', '_', 'x'], ['_', '_', 'x']]
>>> id(board2[0])
139738065760064
>>> id(board2[1])
139738065760064
>>> id(board2[2])
139738065760064

which means it just appends the same row three times.

Augmented Assignments (+= and *=)

augmented assignments are NOT the same as adding or multiplying.
While + internaly uses special function __add__(), += uses special function __iadd__().
Only if __iadd__() is not implemented in the class, it works the same as __add__().

so, when does it change?
In mutable sequences (such as lists), __iadd__() will be changed in place (using less memory) but in other cases, it will work as __add__() which means a = a + b.

>>> l = [1,2,3]
>>> id(l)
139738065799168
>>> mul_list = l * 3 # __mul__()
>>> id(mul_list)
139738068947456 # different address
>>> l *= 3 # __imul__()
>>> id(l)
139738065799168 # changed in place

A+= Assignment Puzzler

>>> t = (1, 2, [30, 40])
>>> t[2] += [50, 60]

This is a corner case that I won’t go into details since it seems too peripheral.
To explain breifly, this will both make an error but also augment the list into [30, 40, 50, 60].

• Putting mutable items in tuples is not a good idea.
• Augmented assignment is not an atomic operation—we just saw it throwing an exception after doing part of its job.
• Inspecting Python bytecode is not too difficult, and is often helpful to see what is going on under the hood.

Sorting

There are mainly two built-in sorting functions, which is list.sort and sorted.

list.sort does create a copy, but sort the list in place. It returns None to remind it.

Note: This is a nice pattern to know about Python API conventions: Python usually outputs None to make it clear the function doesn’t create a copy but changes the object in place.

sorted creates a new list and returns it. Therefore, takes additional memory. This also means it can take any kinds of iterable object including immutable ones, such as tuples. Note that it always outputs a new list.

Both of the functions take two arguments which are reversed: bool and key: string.

>>> l = ['a', 'b', 'c']
>>> sorted_l = sorted(l) # example of `sorted`
>>> id(l)
139738067138496
>>> sorted_l
['a', 'b', 'c']
>>> id(sorted_l)
139738065799168 # another array created
>>> l.sort() # example of `list.sort`

Sorting is very helpful because once the sequence is sorted, each elements can be very efficiently searched.

Managing Ordered Sequence with bisect

It is good to keep the sorted sequence since sorting is expensive.
bisect.bisect and bisect.insort helps a much efficient way to search and insert than list functions.

Binary Search Algorithm is provided via bisect module in Python standard library.

Bisect is an efficient way to search and insert elements into a sequence while keeping the order in an ordered sequence.

bisect() returns the location where the element should be inserted in order to maintain the order while insort() finds the location and inserts the element.

a quick minimal example would be:

>>> import bisect
>>> l = [1, 2, 3, 4, 5]
>>> l
[1, 2, 3, 4, 5]
# bisect only tells you the location
>>> bisect.bisect(l, 6)
5
>>> bisect.bisect(l, 3)
3
>>> bisect.bisect(l, 4)
4
# insort actually inserts the array
# this is an in-place exchange so only one block of
# memory will be added
>>> bisect.insort(l, 3)
>>> l
[1, 2, 3, 3, 4, 5]
>>> bisect.bisect(l, 3)
4

When a List is Not the Answer

Arrays

When creating an array,

1. you provide a typecode
2. a letter to determine the underlying C type used to store each item in the array.

For example

>>> from array import array
>>> from random import random
>>> floats = array ('d', (random() for i in range(2**10)))

‘b’: signed char
’d’: double

whatever you assign to, the array will interpret as the type you assigned to it.

Fast loading and saving are available via .frombytes and .tofile
Binary file can be 60x faster than reading numbers in text file.

Memory Views

A memoryview is essentially a generalized NumPy array structure in Python itself (without the math). It allows you to share memory between data-structures (things like PIL images, SQLlite databases, NumPy arrays, etc.) without first copying.

NumPy and SciPy

Deque

deque (not dequeue) stands for ‘double-ended queue’.

Removing items from the middle of a deque is not as fast. It’s optimized for appending and popping from the end. Used in LIFO queue.

deque internally is implemented as a doubly linked list of fixed-size blocks. Therefore traversing to the middle would be more ineffecient than accesing the ends.

Hash tables are the engines behind Python’s high-performance dicts.

All mapping types in the std library use the basic dict in their implementation, which means all keys(while values are not required) must be hashable.

This means, in order to understand how dictionary or set in Python works under the hood, we should understand about hash first.

About hash

What is hash and what does it mean to be hashable?

The hash() built-in function works directly with built-in types and falls back to calling hash for user-defined types. If two objects compare equal, their hash values must also be equal, otherwise the hash table algorithm does not work.

A hash is a numeric value that Python calculates for a hashable object.
It’s like a unique, short fingerprint for that specific object.
hashable means it can be hashed.

Why does hash make searching faster?

When we feed in a hashable key-value pair, Python will take the (hashable) key and feed it to a hash function. This function will quickly compute an integer number called hash value.

This hash value is then mathematically mapped to a specific index (or bucket) within an internal array that the dictionary uses to store its items.

This makes direct memory access available. This means, hash table design allows Python to use key to go directly to the calculated index in its internal value, without needing to scan other parts of the memory.

What are the hashable types?

• atomic immutable types (such as str)
• frozen set
• tuple is hashable only if all its items are hashable
  • If we look at the example below, unlike tf, tl will produce an error since there is an unhashable list inside the tuples.

>>> tl = (1,2,[30,40])
>>> hash(tl)
Traceback (most recent call last): 
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>>
>>> tf = (1,2,frozenset([30,40])) 
>>> hash(tf)
5149391500123939311

What is a hash table?

Hash table is a sparse array. Cells in hash table are often called buckets
Python tries to keep at least 1/3 of the buckets empty; if the hash table becomes too crowded, it is copied to a new location with room for more buckets (dynamic memory allocation)

When different hash values result in the same bucket, it is called hash collision. There are several ways to deal with hash collision. Python uses open addressing which uses a complex internal algorithm to assign the next bucket. The main goal is to reduce hash collision as much as possible.

set vs frozenset
The difference is mutability.
frozenset is an immutable version of set. You can’t add, remove, or change its elements.

Variations of dicts

There are multiple ways to create dicts.

>>> a = dict(one=1, two=2, three=3)
>>> b = {'one': 1, 'two': 2, 'three': 3}
# combine multiple iterable objects element-wise
>>> c = dict(zip(['one', 'two', 'three'], [1,2,3]))
>>> d = dict([('two',2), ('one', 1), ('three', 3)])
>>> e = dict({'three': 3, 'one': 1, 'two': 2})
>>> a == b == c == d == e  
True

Handling Missing Keys with setdefault

When looking up table and checking if it’s key is missing, it’s more efficient to use setdefault than d.get(k, default).

Mappings with Flexible Key Lookup

This is used to return some made up value when missing key is searched.

We can do this either by 1. default dict instead of dict or 2. any other mapping type + __missing__ method.

defaultdict

index = collections.defaultdict(list)

if default factory is not provided, KeyError will be raised for missing keys.

Variations of dict

collections.OrderedDict
Maintains keys in insertion order
FIFO when it’s default, but LIFO if my_odict.popitem(last=True)

collections.ChainMap

dict and set under the hood

list vs set/dict
because there is no hash table to support searches with the in operator on a list, so a full scan must be made, resulting in times that grow linearly with the size of the haystack.

dicts have significant memory overhead

When you use dictionaries, Python has to do extra work behind the scenes to make them fast for looking things up. This extra work involves creating a “hash table,” which is like a special, spacious grid for organizing data.

hash tables must be sparse to work, which means it’s NOT space efficient

If you have many dictionaries, you are using memory-intensive hash tables. Tuples might be a better choice to avoid significant memory overhead.

Key Search is very fast

dict is trading space for time. They provide fast access regardless of the size of the dict.

When hash collisions occur, whether a bumps b to a different memory location (A-B case) or b bumps a (B-A case), the key point is while Python has enough memory space to handle these adjustments, they are considered the same.

Python’s ==(equality) check for dictionaries focuses solely on whether the same key-value pairs exist in both dictionaries. It doesn’t care about the internal order of keys or the specific memory addresses where they’re stored. As long as key_A maps to value_A and key_B maps to value_B in both dictionaries, they are considered equal, regardless of the behind-the-scenes memory acrobatics.

Adding items to a dict may change the order of existing keys

Dynamic memory allocation may happen as more items are added to the hash table (in order to keep it sparse). During this process, new but different hash collisions may happen while copying the existing keys which changes the order of them.

This is why modifying the contents of a dict while iterating is generally a bad idea.
If you need to scan and add items to a dictionary, first read all of the elements (without modification) and collect the needed additions in a second dict. Then update the original dict by appending the new one.

How Sets work

set is similar to dict but the only difference is that each bucket holds only a reference to the element, while dict holds both key and value.

Basically, set is a dictionary but does not have value attached.
Only keys.

As mentioned in page 97 (“In the end, most of the issues covered in this chapter do not affect programmers who deal only with ASCII text.”), I will skip most of the parts in this chapter since these does not seem to be my immediate interest for now.

Byte Essentials

two types of binary sequences:

• mutable bytearray
• immutable bytes

>>> cafe = bytes('café', encoding='utf-8')
>>> cafe
b'caf\xc3\xa9'
>>> cafe[0]
99
>>> cafe[:1]
b'c'

bytes can be built from a str, given an encoding.
All the elements in utf-8 encoded bytes are ints of range(256) (0~256).
Check the Note below why cafe[0] is int but cafe[:1] isn’t.

Note: The fact that cafe[0] retrieves an int but cafe[:1] returns a bytes object of length 1 should not be surprising. The only sequence type where s[0] == s[:1] is the str type. Al though practical, this behavior of str is exceptional. For every other sequence, s[i] returns one item, and s[i:i+1] returns a sequence of the same type with the s[1] item inside it

Handling Text Files

The best pracitce of handling text file is to follow the “Unicode sandwitch” principle. It means that bytes should be decoded to str as early as possible on input, processed as str, and encoded to bytes as late as possible.

Part 3. Functions as Objects

Chapter 5. First-Class Function

Functions in Python are first-class objects.

Programming language theorists define a “first-class object” as a program entity that can be:
• Created at runtime
• Assigned to a variable or element in a data structure
• Passed as an argument to a function
• Returned as the result of a function

Treating Function Like an Object

Example:

>>> def factorial(n):
...     '''returns n'''
...     return 1 if n < 2 else n * factorial(n-1)
...
>>> factorial(42)
1405006117752879898543142606244511569936384000000000
>>> factorial
<function factorial at 0x7f66ffe57880>
>>> fact = factorial
>>> fact
<function factorial at 0x7f66ffe57880>
>>> fact(42)
1405006117752879898543142606244511569936384000000000
>>> list(map(fact, range(11)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

Higher-Order Functions

Higher-Order Function is function that takes a function as argument or returns function as the result. (ex. map, filter, reduce, sorted …)

For example, we can take functions as elements for the key= parameter in sorted function:

>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'rasberry', 'banana']  

# using len
>>> sorted(fruits, key=len)  
['fig', 'apple', 'cherry', 'banana', 'rasberry', 'strawberry']

# using custom function(reverse)
>>> def reverse(word):  
... return word[::-1]
>>> sorted(fruits, key=reverse)  
['banana', 'apple', 'fig', 'rasberry', 'strawberry', 'cherry']  

Note that key is only used to determine the sorting order, but does NOT change the content.

Modern Replacements for map, filter, and reduce

We can use listcomp and genexp instead of map.
For example,

# map and  listcomp
>>> a1 = list(map(factorial, range(6)))  
>>> a2 = [factorial(x) for x in range(6)]  
>>> a1  
[1, 1, 2, 6, 24, 120]  
>>> a2  
[1, 1, 2, 6, 24, 120]  

map vs generator

map and generator are two different types of iterators.

>>> a3 = map(factorial, range(6))  
>>> a4 = (factorial(x) for x in range(6)
>>> type(a3)  
<class 'map'>  
>>> type(a4)  
<class 'generator'>