Fast Python

Name: Fast Python | High performance techniques for large datasets
Brand: Simon + Schuster LLC
Price: 49.44 EUR
Availability: OnlineOnly

High performance techniques for large datasets

Tiago Antao(Author)

Simon + Schuster LLC (Publisher)

Published on 4. July 2023

304 pages

E-Book

ePUB with Adobe-DRM

System requirements

978-1-63835-686-8 (ISBN)

€49.44incl. 7% vat

System requirements

for ePUB with Adobe-DRM

E-Book Single Licence

Available for download

Description

Alles über E-Books | Antworten auf Fragen rund um E-Books, Kopierschutz und Dateiformate finden Sie in unserem Info- & Hilfebereich.

Alles über E-Books, Kopierschutz & Dateiformate finden Sie in unserem Info- & Hilfebereich.

Master Python techniques and libraries to reduce run times, efficiently handle huge datasets, and optimize execution for complex machine learning applications. Fast Python is a toolbox of techniques for high performance Python including:

Writing efficient pure-Python code
Optimizing the NumPy and pandas libraries
Rewriting critical code in Cython
Designing persistent data structures
Tailoring code for different architectures
Implementing Python GPU computing

Fast Python is your guide to optimizing every part of your Python-based data analysis process, from the pure Python code you write to managing the resources of modern hardware and GPUs. You'll learn to rewrite inefficient data structures, improve underperforming code with multithreading, and simplify your datasets without sacrificing accuracy. Written for experienced practitioners, this book dives right into practical solutions for improving computation and storage efficiency. You'll experiment with fun and interesting examples such as rewriting games in Cython and implementing a MapReduce framework from scratch. Finally, you'll go deep into Python GPU computing and learn how modern hardware has rehabilitated some former antipatterns and made counterintuitive ideas the most efficient way of working. About the Technology Face it. Slow code will kill a big data project. Fast pure-Python code, optimized libraries, and fully utilized multiprocessor hardware are the price of entry for machine learning and large-scale data analysis. What you need are reliable solutions that respond faster to computing requirements while using less resources, and saving money. About the Book Fast Python is a toolbox of techniques for speeding up Python, with an emphasis on big data applications. Following the clear examples and precisely articulated details, you'll learn how to use common libraries like NumPy and pandas in more performant ways and transform data for efficient storage and I/O. More importantly, Fast Python takes a holistic approach to performance, so you'll see how to optimize the whole system, from code to architecture. What's Inside

Rewriting critical code in Cython
Designing persistent data structures
Tailoring code for different architectures
Implementing Python GPU computing

About the Reader For intermediate Python programmers familiar with the basics of concurrency. About the Author Tiago Antão is one of the co-authors of Biopython, a major bioinformatics package written in Python. Table of Contents: PART 1 - FOUNDATIONAL APPROACHES 1 An urgent need for efficiency in data processing 2 Extracting maximum performance from built-in features 3 Concurrency, parallelism, and asynchronous processing 4 High-performance NumPy PART 2 - HARDWARE 5 Re-implementing critical code with Cython 6 Memory hierarchy, storage, and networking PART 3 - APPLICATIONS AND LIBRARIES FOR MODERN DATA PROCESSING 7 High-performance pandas and Apache Arrow 8 Storing big data PART 4 - ADVANCED TOPICS 9 Data analysis using GPU computing 10 Analyzing big data with Dask

More details

Other editions

Content

Intro
inside front cover
Fast Python
Copyright
contents
front matter
preface
acknowledgments
about this book
Who should read this book?
How this book is organized: A road map
About the code
liveBook discussion forum
Hardware and software
about the author
about the cover illustration
Part 1. Foundational Approaches
1 An urgent need for efficiency in data processing
1.1 How bad is the data deluge?
1.2 Modern computing architectures and high-performance computing
1.2.1 Changes inside the computer
1.2.2 Changes in the network
1.2.3 The cloud
1.3 Working with Python's limitations
1.3.1 The Global Interpreter Lock
1.4 A summary of the solutions
Summary
2 Extracting maximum performance from built-in features
2.1 Profiling applications with both IO and computing workloads
2.1.1 Downloading data and computing minimum temperatures
2.1.2 Python's built-in profiling module
2.1.3 Using local caches to reduce network usage
2.2 Profiling code to detect performance bottlenecks
2.2.1 Visualizing profiling information
2.2.2 Line profiling
2.2.3 The takeaway: Profiling code
2.3 Optimizing basic data structures for speed: Lists, sets, and dictionaries
2.3.1 Performance of list searches
2.3.2 Searching using sets
2.3.3 List, set, and dictionary complexity in Python
2.4 Finding excessive memory allocation
2.4.1 Navigating the minefield of Python memory estimation
2.4.2 The memory footprint of some alternative representations
2.4.3 Using arrays as a compact representation alternative to lists
2.4.4 Systematizing what we have learned: Estimating memory usage of Python objects
2.4.5 The takeaway: Estimating memory usage of Python objects
2.5 Using laziness and generators for big-data pipelining
2.5.1 Using generators instead of standard functions
Summary
3 Concurrency, parallelism, and asynchronous processing
3.1 Writing the scaffold of an asynchronous server
3.1.1 Implementing the scaffold for communicating with clients
3.1.2 Programming with coroutines
3.1.3 Sending complex data from a simple synchronous client
3.1.4 Alternative approaches to interprocess communication
3.1.5 The takeaway: Asynchronous programming
3.2 Implementing a basic MapReduce engine
3.2.1 Understanding MapReduce frameworks
3.2.2 Developing a very simple test scenario
3.2.3 A first attempt at implementing a MapReduce framework
3.3 Implementing a concurrent version of a MapReduce engine
3.3.1 Using concurrent.futures to implement a threaded server
3.3.2 Asynchronous execution with futures
3.3.3 The GIL and multithreading
3.4 Using multiprocessing to implement MapReduce
3.4.1 A solution based on concurrent.futures
3.4.2 A solution based on the multiprocessing module
3.4.3 Monitoring the progress of the multiprocessing solution
3.4.4 Transferring data in chunks
3.5 Tying it all together: An asynchronous multithreaded and multiprocessing MapReduce server
3.5.1 Architecting a complete high-performance solution
3.5.2 Creating a robust version of the server
Summary
4 High-performance NumPy
4.1 Understanding NumPy from a performance perspective
4.1.1 Copies vs. views of existing arrays
4.1.2 Understanding NumPy's view machinery
4.1.3 Making use of views for efficiency
4.2 Using array programming
4.2.1 The takeaway
4.2.2 Broadcasting in NumPy
4.2.3 Applying array programming
4.2.4 Developing a vectorized mentality
4.3 Tuning NumPy's internal architecture for performance
4.3.1 An overview of NumPy dependencies
4.3.2 How to tune NumPy in your Python distribution
4.3.3 Threads in NumPy
Summary
Part 2. Hardware
5 Re-implementing critical code with Cython
5.1 Overview of techniques for efficient code re-implementation
5.2 A whirlwind tour of Cython
5.2.1 A naive implementation in Cython
5.2.2 Using Cython annotations to increase performance
5.2.3 Why annotations are fundamental to performance
5.2.4 Adding typing to function returns
5.3 Profiling Cython code
5.3.1 Using Python's built-in profiling infrastructure
5.3.2 Using line_profiler
5.4 Optimizing array access with Cython memoryviews
5.4.1 The takeaway
5.4.2 Cleaning up all internal interactions with Python
5.5 Writing NumPy generalized universal functions in Cython
5.5.1 The takeaway
5.6 Advanced array access in Cython
5.6.1 Bypassing the GIL's limitation on running multiple threads at a time
5.6.2 Basic performance analysis
5.6.3 A spacewar example using Quadlife
5.7 Parallelism with Cython
Summary
6 Memory hierarchy, storage, and networking
6.1 How modern hardware architectures affect Python performance
6.1.1 The counterintuitive effect of modern architectures on performance
6.1.2 How CPU caching affects algorithm efficiency
6.1.3 Modern persistent storage
6.2 Efficient data storage with Blosc
6.2.1 Compress data
save time
6.2.2 Read speeds (and memory buffers)
6.2.3 The effect of different compression algorithms on storage performance
6.2.4 Using insights about data representation to increase compression
6.3 Accelerating NumPy with NumExpr
6.3.1 Fast expression processing
6.3.2 How hardware architecture affects our results
6.3.3 When NumExpr is not appropriate
6.4 The performance implications of using the local network
6.4.1 The sources of inefficiency with REST calls
6.4.2 A naive client based on UDP and msgpack
6.4.3 A UDP-based server
6.4.4 Dealing with basic recovery on the client side
6.4.5 Other suggestions for optimizing network computing
Summary
Part 3. Applications and Libraries for Modern Data Processing
7 High-performance pandas and Apache Arrow
7.1 Optimizing memory and time when loading data
7.1.1 Compressed vs. uncompressed data
7.1.2 Type inference of columns
7.1.3 The effect of data type precision
7.1.4 Recoding and reducing data
7.2 Techniques to increase data analysis speed
7.2.1 Using indexing to accelerate access
7.2.2 Row iteration strategies
7.3 pandas on top of NumPy, Cython, and NumExpr
7.3.1 Explicit use of NumPy
7.3.2 pandas on top of NumExpr
7.3.3 Cython and pandas
7.4 Reading data into pandas with Arrow
7.4.1 The relationship between pandas and Apache Arrow
7.4.2 Reading a CSV file
7.4.3 Analyzing with Arrow
7.5 Using Arrow interop to delegate work to more efficient languages and systems
7.5.1 Implications of Arrow's language interop architecture
7.5.2 Zero-copy operations on data with Arrow's Plasma server
Summary
8 Storing big data
8.1 A unified interface for file access: fsspec
8.1.1 Using fsspec to search for files in a GitHub repo
8.1.2 Using fsspec to inspect zip files
8.1.3 Accessing files using fsspec
8.1.4 Using URL chaining to traverse different filesystems transparently
8.1.5 Replacing filesystem backends
8.1.6 Interfacing with PyArrow
8.2 Parquet: An efficient format to store columnar data
8.2.1 Inspecting Parquet metadata
8.2.2 Column encoding with Parquet
8.2.3 Partitioning with datasets
8.3 Dealing with larger-than-memory datasets the old-fashioned way
8.3.1 Memory mapping files with NumPy
8.3.2 Chunk reading and writing of data frames
8.4 Zarr for large-array persistence
8.4.1 Understanding Zarr's internal structure
8.4.2 Storage of arrays in Zarr
8.4.3 Creating a new array
8.4.4 Parallel reading and writing of Zarr arrays
Summary
Part 4. Advanced Topics
9 Data analysis using GPU computing
9.1 Making sense of GPU computing power
9.1.1 Understanding the advantages of GPUs
9.1.2 The relationship between CPUs and GPUs
9.1.3 The internal architecture of GPUs
9.1.4 Software architecture considerations
9.2 Using Numba to generate GPU code
9.2.1 Installation of GPU software for Python
9.2.2 The basics of GPU programming with Numba
9.2.3 Revisiting the Mandelbrot example using GPUs
9.2.4 A NumPy version of the Mandelbrot code
9.3 Performance analysis of GPU code: The case of a CuPy application
9.3.1 GPU-based data analysis libraries
9.3.2 Using CuPy: A GPU-based version of NumPy
9.3.3 A basic interaction with CuPy
9.3.4 Writing a Mandelbrot generator using Numba
9.3.5 Writing a Mandelbrot generator using CUDA C
9.3.6 Profiling tools for GPU code
Summary
10 Analyzing big data with Dask
10.1 Understanding Dask's execution model
10.1.1 A pandas baseline for comparison
10.1.2 Developing a Dask-based data frame solution
10.2 The computational cost of Dask operations
10.2.1 Partitioning data for processing
10.2.2 Persisting intermediate computations
10.2.3 Algorithm implementations over distributed data frames
10.2.4 Repartitioning the data
10.2.5 Persisting distributed data frames
10.3 Using Dask's distributed scheduler
10.3.1 The dask.distributed architecture
10.3.2 Running code using dask.distributed
10.3.3 Dealing with datasets larger than memory
Summary
Appendix A. Setting up the environment
A.1 Setting up Anaconda Python
A.2 Installing your own Python distribution
A.3 Using Docker
A.4 Hardware considerations
Appendix B. Using Numba to generate efficient low-level code
B.1 Generating optimized code with Numba
B.2 Writing explicitly parallel functions in Numba
B.3 Writing NumPy-aware code in Numba
index

System requirements

Save as PDF Copy link into clipboard

Schweitzer Fachinformationen

Fast Python

Description

More details

Other editions

Additional editions

Content

System requirements