Streamlining Your Research Laboratory with Python
Description
Enables scientists and researchers to efficiently use one of the most popular programming languages in their day-to-day work
Streamlining Your Research Laboratory with Python covers the Python programming language and its ecosystem of tools applied to tasks encountered by laboratory scientists and technicians working in the life sciences. After opening with the basics of Python, the chapters move through working with and analyzing data, generating reports, and automating the lab environment.
The book includes example processes within chapters and code listings on nearly every page along with schematics and plots that can clearly illustrate Python at work in the lab. The book also explores some real-world examples of Python's application in research settings, demonstrating its potential to streamline processes, improve productivity, and foster innovation.
Streamlining Your Research Laboratory with Python includes information on:
- Language basics including the interactive console, data types, variables and literals, strings, and expressions using operators
- Custom functions and exceptions such as arguments and parameters, names and scope, and decorators
- Conditional and repeated execution as methods to control the flow of a program
- Tools such as JupyterLab, Matplotlib, NumPy, pandas DataFrame, and SciPy
- Report generation in Microsoft Word and PowerPoint, PDF report generation, and serving results through HTTP and email automatically
Whether you are a biologist analyzing genetic data, a chemist scouting synthesis routes, an engineer optimizing machine parameters, or a social scientist studying human behavior, Streamlining Your Research Laboratory with Python serves as a logical and practical guide to add Python to your research toolkit.
More details
Other editions
Additional editions
Persons
Mark F. Russo, PhD is currently on the faculty in the Department of Computer Science at The College of New Jersey. Previously, he had a multi-decade professional career in biotech and pharma with a focus on scientific computing, automation, and scientific data.
William Neil is currently at Bristol Myers Squibb and has been working in the pharmaceutical industry since 1995.
Content
Preface xv
1 Introduction 1
1.1 Python Implementations 1
1.2 Installing the Python Toolkit 2
1.3 Python 3 vs. Python 2 2
1.4 Python Package Index 3
1.5 Programming Editors 5
1.6 Notebook Editors 5
1.7 Using the Jupyter Notebook Interface 7
1.8 JupyterLite 8
1.9 Things Change 9
1.10 Key Takeaways 9
2 Language Basics 11
2.1 Python Interactive Console 11
2.2 Data Types 11
2.3 Variables and Literals 14
2.4 Strings 15
2.4.1 Simple Strings 15
2.4.2 Multi-Line Strings 16
2.4.3 Escape Characters in a String 16
2.4.4 Raw Strings 18
2.4.5 Formatted Strings 18
2.4.6 Strings as Objects 19
2.4.7 Characters and Encodings 21
2.5 Expressions Using Operators 22
2.5.1 Arithmetic Operators 22
2.5.2 Assignment Operators 24
2.5.3 Comparison Operators 25
2.5.4 Boolean Operators 26
2.5.5 Chaining Comparisons 28
2.5.6 Comparing Floating-Point Numbers 29
2.6 Functions and How to Use Them 29
2.6.1 Invoking Functions 30
2.6.2 Built-in Functions 31
2.6.3 The math Module for Additional Mathematical Functions 31
2.6.4 The random Module for Pseudo-Random Number Generation 33
2.6.5 The time and datetime Modules for Handling Dates and Times 36
2.6.6 The sys Module for System Interactions 39
2.6.7 Scope and Namespace 40
2.7 Your First Python Program 41
2.8 Key Takeaways 42
3 Data Structures 45
3.1 Lists 45
3.1.1 Introducing Lists 45
3.1.2 Global Functions That Operate on Lists 46
3.1.3 Accessing List Elements 46
3.1.4 Slicing Lists 47
3.1.5 Lists Operators 49
3.1.6 Lists as Objects 51
3.2 Tuples 55
3.2.1 Introducing Tuples 55
3.3 Dictionaries 56
3.3.1 Introducing Dictionaries 56
3.3.2 Global Functions That Operate on Dictionaries 57
3.3.3 Accessing Dictionary Items 57
3.3.4 Dictionary Operators 58
3.3.5 Dictionaries as Objects 59
3.4 Sets 61
3.4.1 Introducing Sets 61
3.4.2 Global Functions That Operate on Sets 62
3.4.3 Accessing Set Elements 62
3.4.4 Set Operators 62
3.4.5 Sets as Objects 64
3.5 Destructuring Assignment 65
3.6 Key Takeaways 66
4 Controlling the Flow of a Program 67
4.1 Conditional Execution 67
4.1.1 If-Statements 67
4.1.2 If-Else Statements 68
4.1.3 If-Elif-Else Statements 70
4.1.4 If-Statement Strategies 71
4.1.5 Truthy and Falsy Values 74
4.1.6 Conditional Expressions 75
4.2 Repeated Execution 76
4.2.1 While-Statements 76
4.2.2 For-Statements 81
4.2.3 For-Statements with Range 83
4.2.4 Break and Continue 84
4.2.5 Comprehensions 85
4.3 Key Takeaways 87
5 Custom Functions and Exceptions 89
5.1 Defining Custom Functions 89
5.2 Arguments and Parameters 94
5.3 Names and Scope 97
5.3.1 Local vs. Global 97
5.3.2 Built-in and Nonlocal Scope 99
5.4 Scope vs. Namespace 100
5.5 Organizing Your Code with Modules 102
5.6 Decorators 104
5.7 How Things Go Wrong 106
5.8 Python Exceptions 106
5.9 Handling Exceptions 107
5.10 Raising Your Own Exceptions 110
5.11 Key Takeaways 112
6 Regular Expressions 115
6.1 Matching Literal Text 115
6.2 Alternation 116
6.3 Defining and Matching Character Classes 116
6.4 Metaclasses 117
6.5 Pattern Sequences 117
6.6 Repeating Patterns with Quantifiers 118
6.7 Anchors 119
6.8 Capturing Groups 120
6.9 Regular Expressions in Python 122
6.10 Project - A Formula Mass Calculator 123
6.11 Key Takeaways 130
7 Working with Data 131
7.1 A File System Primer 131
7.2 Text Files 133
7.3 Reading and Writing Text Files 134
7.4 Working with Comma-Separated Values (CSV) Files 139
7.5 The csv Module 144
7.6 Reading and Writing Excel Spreadsheet 145
7.6.1 openpyxl Workbook Object 146
7.6.2 openpyxl Worksheet Object 147
7.6.3 openpyxl Cell Object 148
7.7 Project - Generate a Random Sample Layout in a Spreadsheet 148
7.8 Project - Forecast Monthly Sample Processing 150
7.9 Managing the File System 154
7.9.1 The Path Object 155
7.9.2 Path Properties 155
7.9.3 Path Attributes 156
7.9.4 Operating On a Path 157
7.9.5 Combining Paths 158
7.9.6 The shutil Module for High-Level File Operations 159
7.10 Walking a File System Tree 159
7.11 Project - Find Duplicate Files 161
7.12 Working with Zip Files 163
7.12.1 ZipFile Object 163
7.12.2 zipfile Path Object 163
7.12.3 Creating Zip Archives 164
7.13 Working with Standard Data Formats 165
7.13.1 JSON - JavaScript Object Notation 165
7.13.2 json Python Module 166
7.13.3 XML - Extensible Markup Language 168
7.13.4 Python XML modules 170
7.13.5 Other Standard Data Formats 176
7.14 Key Takeaways 176
8 Web Resources 179
8.1 TCP/IP Networks - What You Need to Know 179
8.1.1 Internet Protocol 180
8.1.2 Transmission Control Protocol 180
8.1.3 Connections and Ports 180
8.1.4 Application Layer Protocols 181
8.1.5 IPv4 vs. IPv6 Addresses 181
8.1.6 Proxy Servers 181
8.2 Introduction to Hypertext Transfer Protocol 182
8.2.1 The Uniform Resource Locator 183
8.2.2 Anatomy of an HTTP Request 184
8.2.3 Anatomy of an HTTP Response 186
8.3 Web Services and the Python Requests Module 187
8.3.1 HTTP GET Requests and the Response Object 188
8.3.2 HTTP POST Requests 189
8.3.3 Binary Responses 190
8.3.4 Customizing the Request Object 193
8.3.5 Verifying Certificates and Encryption 193
8.3.6 Other requests Module Options 194
8.4 Project - Print Weather Forecast for a Location 195
8.4.1 National Weather Service API Web Service 195
8.4.2 Getting Forecast URL from Geolocation 197
8.4.3 Loading and Processing Forecast Data 197
8.4.4 Completed Program to Generate Temperature Forecast 198
8.5 Project - Scraping HTML Page Content 201
8.6 Key Takeaways 206
9 Data Analysis and Visualization 209
9.1 JupyterLab 209
9.2 Scientific Plotting with Matplotlib 211
9.2.1 The pyplot Submodule 212
9.2.2 The pyplot.plot() function 212
9.2.3 Customizing a Plot 213
9.2.4 Multiple Curves on a Single Plot 214
9.2.5 Additional Plot Types 214
9.2.6 Multiple Axes on a Single Figure 214
9.2.7 Other Useful Functions 216
9.2.8 Project - Plotting Weather Forecast 216
9.2.9 Project - A Custom Microplate Heat Map 219
9.2.10 Other Scientific Plotting Libraries 222
9.3 NumPy - Numerical Python 223
9.3.1 Creating ndarray Objects 223
9.3.2 Working with ndarray Objects 223
9.3.3 Accessing and Updating ndarray Elements 224
9.3.4 Broadcasting 226
9.4 pandas DataFrame 226
9.4.1 Creating and Inspecting DataFrames 227
9.4.2 Filtering DataFrames 228
9.4.3 Project - A Screening Experiment 230
9.5 SciPy - A Library for Mathematics, Science, and Engineering 239
9.5.1 Descriptive Statistics with SciPy 239
9.5.2 Hypothesis Testing 239
9.5.3 Project - Running Hypothesis Tests on Two Samples 240
9.5.4 Project - Comparing Liquid Handler Syringe Performance 243
9.5.5 Linear Regression 245
9.5.6 Fitting Nonlinear Models to Data 246
9.5.7 Project - Four-Parameter Logistic Regression 248
9.6 Key Takeaways 253
10 Report Generation 255
10.1 BytesIO Object 255
10.2 Generating Reports in Microsoft Word 257
10.2.1 Document Object 257
10.2.2 Paragraph Object 258
10.2.3 Run Object 259
10.2.4 Picture and InlineShape Objects 260
10.2.5 Table Object 261
10.2.6 Project - Generate a Complete Word Report 263
10.3 Generating Microsoft PowerPoint Presentations 266
10.3.1 Presentation Object 266
10.3.2 Slide Objects 267
10.3.3 SlideShapes Object 268
10.3.4 Length Objects 269
10.3.5 Table Object 269
10.3.6 Project - Generate a PowerPoint Document with Figures and Tables 272
10.4 Generating PDF File Reports 275
10.4.1 ReportLab PDF Generation Process 275
10.4.2 Creating a Canvas Object 276
10.4.3 Setting Canvas Styles 276
10.4.4 Managing Text Blocks with PDFTextObjects 278
10.4.5 Canvas State Stack 281
10.4.6 Drawing Images 281
10.4.7 PLATYPUS for Page Layout 283
10.4.8 Project - Generate a Complete PDF Report 283
10.5 Sending Email Programmatically 287
10.5.1 Simple Mail Transfer Protocol 288
10.5.2 SMTP Mail Server 288
10.5.3 Send a Simple Email Message 288
10.5.4 Sending Email Messages over a Secure Connection 291
10.5.5 Building an Email Message with Attachments 292
10.6 Serving Results with an HTTP Server 294
10.7 Key Takeaways 296
11 Control and Automation 297
11.1 Concurrency in Python 297
11.2 Asynchronous Execution 299
11.3 Concurrent Programs with AsyncIO 300
11.4 Asynchronous Instrument Control and Coordination 304
11.4.1 Project - Integrated Laboratory System Control and Coordination 304
11.5 Communicating over a Serial Port 311
11.5.1 Reading Barcodes from a Serial Port 312
11.5.2 Project - Scanning Sample Tasks into a Running Controller 318
11.6 Execute Remote Commands over HTTP 319
11.6.1 A Basic HTTP Server with aiohttp 320
11.6.2 Routing an HTTP Request to a Custom Python Function 321
11.7 Persistent Network Connections using a WebSocket 325
11.7.1 A User Interface for Asynchronous Networked Programs 326
11.7.2 WebSocketResponse and FileResponse Objects 326
11.7.3 Project - A Browser-Based WebSocket Message Broadcaster 327
11.7.4 Project - A Browser User Interface to Schedule Samples for Analysis 335
11.8 Responding to File System Changes 338
11.8.1 Watching a Directory for Changes with watchfiles 338
11.8.2 File System Monitoring Options 340
11.9 Executing Tasks on a Schedule 341
11.9.1 sched Module 342
11.9.2 Project - Taking and Sending Images on a Schedule 344
11.10 Key Takeaways 348
Postface 351
References 353
Appendix A: ASCII American Standard Code for Information Interchange 357
Index 359
Chapter 1
Introduction
Python is one of the most popular programming languages, and for good reasons. Among the principles that guide the design of the language, called the Zen of Python, is the principle that Readability Counts. You will discover this repeatedly throughout your learning journey. Unlike source code that you may have encountered in the past written in other programming languages, Python will not appear to the untrained eye as an undiscovered form of hieroglyphics. If written well, Python source code can be relatively easy to read and understand, and it can be equally straightforward to write. Perhaps this is what has driven the popularity of Python.
As a general-purpose programming language, you will find that learning Python provides you with the power to solve virtually any computing problem that you may encounter. This includes data collection and processing, instrument control, scientific computations, publication quality graphing, report generation, and much more. Many of these packages are designed to solve the kinds of scientific problems encountered in a research laboratory. We will cover many of the most popular and widely used scientific Python packages. When combining Python's clean and simple syntax with over 600,000 packages that are freely available in the Python Package Index (PyPI), you will find that you have at your fingertips an incredibly powerful toolkit to solve authentic scientific and research laboratory problems.
Another incredible feature of Python is that it has been selected as one of the two most popular languages used for Data Science. As a research scientist, experimental data is likely the lens through which you learn about the world. While you may never need the full range of advanced numerical, statistical, and modeling features required by a professional data scientist, it is no doubt that you will benefit from the power of Python to operate legitimately in the Data Science field.
Finally, it is worth noting that Python is an open-source language, which means it is free to use and has a large and active community of developers. This community continuously maintains and improves Python's vast array of libraries and other tools, making it a robust platform for scientific research. Python is available for most operating systems, which ensures that you will be able to run it wherever you need it, even on microcontrollers.
1.1 Python Implementations
Although not formally standardized like other programming languages such as C, C++, and JavaScript, the Python language syntax is defined by The Python Language Reference [1] as well as its reference implementation in C called CPython, which is available for all major operating systems. Both the language reference documentation and CPython implementation are available from Python's official website at https://www.python.org [2].
Python may be implemented by anyone on any platform. Consequently, and due to its significant popularity, you will find Python available on almost every computing platform. In addition to CPython, which is available for all major operating systems, including Windows, MacOS, and Linux, an implementation of Python called IronPython has been implemented for the .Net runtime [3], Jython has been implemented for the Java Virtual Machine (JVM) [4], and at least two Python subsets that run on microcontrollers: MicroPython [5] and CircuitPython [6]. A version of Python has even been implemented in Python itself, appropriately called PyPy [7].
More recently, CPython and many of its most important packages have been compiled to WebAssembly [8]. WebAssembly is a stack-based virtual machine that runs entirely in and is confined by, a web browser. The WebAssembly port of Python is called Pyodide [9]. Pyodide allows us to run Python in a web browser without the need to install it first. Pyodide is an important option for laboratory scientists because laboratory computers are often locked down for security reasons. Typically, the primary purpose of a lab computer is to operate an attached instrument or process data, not to perform general-purpose computing. For security reasons, it is often the case that installing new software is strictly forbidden. Fortunately, because the entire Pyodide Python environment may be loaded into a web browser directly, Pyodide provides a means to access the power of Python without the need to convince your IT team to grant you the elevated privileges required to install software.
No matter which implementation of Python you choose, your knowledge of the Python programming language will be instrumental in helping you streamline your laboratory operations.
1.2 Installing the Python Toolkit
To install CPython, visit the official Python home page at https://www.python.org/ and click the Downloads link [2]. An appropriate installer for your operating system will be offered. Download and run the installer. Python will be installed for you on your computer.
To test your installation, open a terminal program and enter the python command. Many terminal programs are available and will change based upon your computer's operating system. On MacOS and Linux, you should find Terminal as one of your program options. On Windows, you may use PowerShell, Command Prompt, or another option. But no matter which terminal program you use, simply enter the python or python3 command into the terminal program and press Enter. This command will run the Python Interactive Console, which you may use to execute Python commands interactively.
To exit the Python console, enter the Python command exit(). See Figure 1.1 for an example.
Figure 1.1 The Python Interactive Console running in Windows PowerShell.
If the installation of Python is successful but you are still having problems starting the Python console, our experience suggests that the problem lies with your operating system's ability to find the Python executable. Investigate where Python was installed and make sure that the path location is included in your PATH environment variable. Also check that you have the necessary permissions to access and execute the python program.
1.3 Python 3 vs. Python 2
Python 3 was introduced to the community in 2008 as a "breaking change" version of Python. Programs written for the previous Python 2 would not run in Python 3 due to significant changes to syntax and other implementation details. This change was necessary because several of the design decisions made for Python 2 needed an upgrade to make the language more suitable for modern applications. Some changes made to Python 3 were fundamental, including the way binary data is stored and processed.
There was a significant number of existing Python 2 programs in production around the world when Python 3 was announced. It is no surprise that many Python 2 programmers were less than enthusiastic about porting their source code to Python 3. Nevertheless, Python 2 was scheduled to be retired in 2015, but the resistance was so strong that this date had to be delayed. It wasn't until January 1, 2020 that Python 2 was finally and fully retired.
Even though Python 2 has been retired and it no longer receives security patches, you can still install and use it. If you have a version of Python 2 installed, please resist the urge to use it to write new programs and install Python 3 instead. For guidance porting Python 2 code to Python 3, refer to Python's own porting guide [10]. If you need both versions of Python available, you may install Python 3 with Python 2 and use them both simultaneously. With both versions of Python installed, run Python 3 from a terminal using the python3 command in place of the python command. To make sure you have a recent version of Python 3 installed, you can enter the following command into a terminal. In this book we use Python 3 exclusively.
python --version 1.4 Python Package Index
One of Python's mottos is "batteries included," and for a good reason. A Python distribution comes with a huge library of prewritten modules for you to use and build upon. While it's true that a Python distribution includes quite a few "batteries," it is not possible to include them all.
If a module is not shipped with Python, there is a good chance someone in the Python community has contributed a module that will help you. Additional Python modules are distributed through a Python package repository, with the two most popular being the PyPI [11] and the Package Repository for Anaconda [12]. Anaconda is an exceptional platform that provides high quality Python installations, package distributions, and other open-source resources. Importantly, it also offers paid support plans, which may be critical for businesses that depend upon Python as part of their core operations. In the following, we describe how to use PyPI for installing additional Python packages.
The PyPI provides a way for package authors to post their open-source Python packages, and for package users to find and install Python packages that are not distributed with Python. The PyPI hosts over 600,000 freely available Python packages ready for you to install and use. If you need something specific, there is a good chance that the PyPI has a package for that.
Python packages are...
