mynameisfiber/high_performance_python

Code for the book "High Performance Python" by Micha Gorelick and Ian Ozsvald with OReilly

High Performance Python: The Code

This repository contains the code from "High Performance Python" by Micha Gorelick and Ian Ozsvald with O'Reilly Media. Each directory contains the examples from the chapter in addition to other interesting code on the subject.

Topics Covered

This book ranges in topic from native Python to external modules to writing your own modules. Code is shown to run on one CPU, multiple coroutines, multiple CPU's and multiple computers. In addition, throughout this exploration a focus is kept on keeping development time fast and learning from profiling output in order to direct optimizations.

The following topics are covered in the code repo:

• Chapter 1: Profiling

  • What are the ways to profile your code?
  • What insights can I gain from a profile?

• Chapter 2: Understanding Performant Programming

  • How does a computer work and how does that affect my code performance?

• Chapter 3: Lists and Tuples

  • How do lists and tuples work?
  • What are the performance implications of this?

• Chapter 4: Dictionaries and Sets

  • How do dictionaries and sets work?
  • What are the performance implications of this?

• Chapter 5: Iterators

  • How do iterators work?
  • How can I use them to reduce memory in my code?
  • How can I use them to reduce the CPU operations needed in my code?

• Chapter 6: Matrix and Vector Computation

  • How does the CPU perform matrix/vector operations?
  • What are the ways I can profile the efficiency of these operations?
  • How can I speed up code given a profile?

• Chapter 7: Compiling to C

  • What are the automated ways to compile my python to C?
  • What are the manual ways to compile my python to C?
  • How can I use these technologies to speed up my code and not slow down development?

• Chapter 8: Concurrency

  • How does concurrency work and how does python support it?
  • How can I speed up multiple IO operations?
  • How can I hide IO wait?

• Chapter 9: Multiprocessing

  • What is multiprocessing?
  • How can I speed up my code on a multiple CPU machine?
  • What are the subtleties of muli-CPU code?

• Chapter 10: Clusters and Job Queues

  • How can I extend my code to a cluster?
  • What are some of the pain-points with clusters?
  • What is a queue and how is it useful?

• Chapter 11: Using Less Ram

  • How can I use algorithms to reduce the RAM usage of my code?
  • What are tries and probabilistic data structures?

• Chapter 12: Lessons from the Field (no code)

  • Some stories from the field on performance python

Using the code base

This code base is a live document and should be freely commented on and used. It is distributed with a license that amounts to: don't use the code for profit, however read the provided license file for the law-jargon. Feel free to share, fork and comment on the code!

If any errors are found, or you have a bone to pick with how we go about doing things, leave an issue on this repo! Just keep in mind that all code was written for educational purposes and sometimes this means favouring readability over "the right thing" (although in Python these two things are generally one and the same!).

high_performance_python's Issues

Recent version of kernprof fails with exception(s)

There are 2 issues:

• print statements syntax no longer support (with either Python 2 or 3);
• Functions have no longer have a func_name attribute;

julia_lineprofiler no working

the example scripts for julia_lineprfiler do not work

thought this does not work in any python version
`
print "Length of x:", len(x)

print "Total elements:", len(zs)

`

@Profile decorator is not explained completely as once added the script can not be called with kernprof

Chapter 4, pg 73

In the following sentence:
While we saw in the previous chapter that we are restricted to, at best, O(log n) lookup
time on lists/tuples with no intrinsic order (through a search operation), dictionaries
and sets give us O(n) lookups based on the arbitrary index.

Shouldn't the bold part part say: O(1) lookups?

Chapter 11: Page 373 Morris counters

The book states: "The maximum we could approximately count where we use a single unsigned byte
for the exponent is math.pow(2,255) == 5e76". The actual value is 5.79 x 10^76, so it should be rounded to 6. This might be intentional as the counters are approximate but I thought I'd bring it up!

Provided noop_profile_decorator does not work properly with line_profiler in Python 3

Hi,
The provided solution for "noop-profile" decorator does not work for Robert Kern's kernprof tool: it fails silently in reaching desired objective, as it prevents exceptions, but also prevent line_profiler from collecting information. It does work for memory_profiler module by Fabian Pedregosa and Philippe Gervais.

I use Python 3.5.2

A solution that works for both line_profiler and memory_profiler is:

# memory profile
try:
    @profile
    def dummy():
        pass
except NameError:
    def profile(func):
        def inner(*args, **kwargs):
            return func(*args, **kwargs)
        return inner

Am I missing something?

Update
Sorry, after reading again this part of the book, I see that two solutions are described, one for line_profiler and one for memory_profiler (the one in the repository that I linked above is the one for memory_profiler). But still, isn't it better to just catch NameError exceptions, rather than having two implementations?

Kind Regards

chapter one questions and answers

1st, what are the elements of a computer architecture?

• memory
• computation unit
• connections

2nd, what are some common alternate computer architectures?
(GPU)???

3rd, how does python abstract the underlying computer architecture?
Python have virtual machine, just like Java have a JVM

4th, what are some of the hurdles to make performant python code?

• Python got a GIL
• Python is a dynamic language
• python's abstraction makes it difficult to do some L1/L2 cache thing

5th, what are different types of performance problems?
???