Web Protocols and Practice
HTTP/1.1, Networking Protocols, Caching, and Traffic Measurement
Published on 29. May 2001
Book
Hardback
672 pages
978-0-201-71088-5 (ISBN)
Description
Just as TCP/IP is a central protocol for the Internet, HTTP is a central protocol for the web. They are both critical for web networking. Web Protocols and Practice is the most authoritative and comprehensive guide to the Web's technical underpinnings. Authored by legendary AT&T Labs researcher Bala Krishnamurthy and renowned Web networking expert Jennifer Rexford, this book offers exceptionally thorough coverage of core Web protocols-including the most detailed discussion of HTTP /1.1 and its relationship to TCP/IP networking ever presented. The authors begin with a broad overview of the evolution of the Web, including its naming infrastructure, HTML document language, and HTTP message exchange protocol. Next, they introduce the inner workings of clients, proxies, and servers, as well as scripts, handlers, search engines, cookies, and authentication. The heart of the book is a detailed discussion of the core Web protocols DNS, TCP/IP, and HTTP. An essential resource for all networking and Internet professionals, and for all developers building Internet applications.
More details
Persons
Balachander Krishnamurthy of AT&T Labs-Research is a well-known researcher and the author of numerous publications in the World Wide Web and networking fields. He has published more than forty-five technical papers, coauthored and edited Practical Reusable UNIX Software, and was the series editor for the Trends in Software series of books. He holds several patents and has been a guest lecturer in more than thirty countries. His recent papers can be found at http://www.research.att.com/~bala/papers.
Jennifer Rexford of AT&T Labs-Research is an established researcher and the author of numerous publications in the areas of IP networking, multimedia streaming, and the World Wide Web. She has served on the editorial board for IEEE/ACM Transactions on Networking and on program committees for the main networking conferences. Her recent papers can be found at http://www.research.att.com/~jrex.
0201710889AB05012001
Jennifer Rexford of AT&T Labs-Research is an established researcher and the author of numerous publications in the areas of IP networking, multimedia streaming, and the World Wide Web. She has served on the editorial board for IEEE/ACM Transactions on Networking and on program committees for the main networking conferences. Her recent papers can be found at http://www.research.att.com/~jrex.
0201710889AB05012001
Content
I. GETTING STARTED.
1. Introduction.
Origin and Growth of the World Wide Web.
Historical Evolution of the Web.
The State of the Web.
Semantic Components of the Web.
Uniform Resource Identifer (URI).
HyperText Markup Language (HTML).
HyperText Transfer Protocol (HTTP).
Terms and Concepts.
Content on the Web.
Software Components.
Underlying Network.
Standardization.
Web.
Web Applications.
Topics Not Covered.
Tour Through the Book.
II. WEB SOFTWARE COMPONENTS.
2. Web Clients.
Client as a program.
Evolution of the browser.
Web-related browser functions.
Canonical Web transfer example.
Issuing a request from a browser.
Browser caching.
Request message headers.
Response handling.
Browser configuration.
Physical appearance.
Semantic choices.
Configuring browser for non-protocol functions.
Browser security issues.
Cookies.
Motivations for cookies.
Use of cookies in a browser.
User's control over cookies.
Privacy problems with cookies.
Spiders.
Searching on the Web.
Spider client.
Use of spiders in search engines.
Intelligent agents and special-purpose browsers.
Intelligent agents.
Special-purpose browsers.
Summary.
3. Web Proxies.
History and evolution of intermediaries.
High-level classification of proxies.
Caching proxy.
Transparent proxy.
Proxy applications.
Sharing access to the Web.
Caching responses.
Anonymizing clients.
Transforming requests and responses.
Gateway to non-HTTP systems.
Filtering requests and responses.
HTTP-related proxy roles.
Steps in request-response exchange with a proxy.
Handling HTTP requests and responses.
Proxy as Web server.
Proxy as Web client.
Example use of a proxy.
Proxy chaining and hierarchies.
Proxy configuration.
Proxy privacy issues.
Other kinds of proxies.
Reverse proxies or surrogates.
Interception proxies.
Summary.
4. Web Servers.
Web Site Versus Web Server.
Web Site.
Web Server.
Handling a Client Request.
Steps in Handling a Client Request.
Access Control.
Dynamically Generated Responses.
Creating and Using Cookies.
Sharing Information Across Requests.
Sharing HTTP Responses Across Requests.
Sharing Metadata Across Requests.
Server Architecture.
Event-Driven Server Architecture.
Process-Driven Server Architecture.
Hybrid Server Architecture.
Server Hosting.
Multiple Web Sites on a Single Machine.
Multiple Machines for a Single Web Site.
Case Study: Apache Web Server.
Resource Management.
HTTP Request Processing.
Summary.
III. WEB PROTOCOLS.
5. Protocols Underlying HTTP.
Internet Protocol.
Evolution of the Internet Architecture.
IP Design Goals.
IP Addresses.
IP Header Details.
Transmission Control Protocol 0.
Socket Abstraction 1.
Ordered Reliable Byte Stream.
Opening and Closing a TCP Connection.
Sliding-Window Flow Control.
Retransmission of Lost Packets.
TCP Congestion Control.
TCP Header Details.
Domain Name System.
DNS Resolver.
DNS Architecture.
DNS Protocol.
DNS Queries and the Web.
DNS-Based Web Server Load Balancing.
Application-Layer Protocols.
Telnet Protocol.
File Transfer Protocol.
Simple Mail Transfer Protocol.
The Network News Transfer Protocol.
Properties of Application-Layer Protocols.
Summary.
6. HTTP Protocol Design and Description.
Overview of HTTP.
Protocol properties.
Protocol infuences.
HTTP language elements.
HTTP terms.
HTTP/1.0 Request Methods.
HTTP/1.0 Headers.
HTTP/1.0 Response classes.
HTTP Extensibility.
SSL and Security.
SSL.
HTTPS: Using SSL in Web exchanges.
Security in HTTP/1.0.
Protocol compliance and interoperability.
Version number and interoperability.
MUST, SHOULD, MAY requirement levels.
Summary.
7. HTTP/1.1 221.
The evolution of HTTP/1.1 protocol.
History of evolution.
Problems with HTTP/1.0 25.
New concepts in HTTP/1.1.
Methods, headers, response codes in 1.0 and 1.1.
Old and new request methods.
Old and new headers.
Old and new response codes.
Caching.
Caching related terms.
Caching in HTTP/1.0.
Caching in HTTP/1.1.
Bandwidth optimization.
The Range request.
The Expect/Continue mechanism.
Compression.
Connection Management.
The Connection: Keep-Alive mechanism of HTTP/1.0.
Evolution of HTTP/1.1 persistent connection mechanism.
The Connection header.
Pipelining on persistent connections.
Closing persistent connections.
Message transmission.
Extensibility.
Learning about the server.
Learning about intermediate servers.
Upgrading to other protocols.
Internet address conservation.
Content negotiation.
Security, authentication, and integrity.
Security and authentication.
Integrity.
The role of proxies in HTTP/1.1.
Types of proxies.
Syntactic requirements on an HTTP/1.1 proxy.
Semantic requirements on an HTTP/1.1 proxy.
Other miscellaneous changes.
Method-related miscellaneous changes.
Header-related miscellaneous changes.
Response-code-related miscellaneous changes.
Summary.
8. HTTP/TCP Interaction.
TCP Timers.
Retransmission Timer.
Slow-Start Restart.
The TIME WAIT State.
HTTP/TCP Layering.
Aborted HTTP Transfers.
Nagle's Algorithm.
Delayed Acknowledgments.
Multiplexing TCP Connections.
Motivation for Parallel Connections.
Problems with Parallel Connections.
Server Overheads.
Combining System Calls.
Managing Multiple Connections.
Summary.
IV. MEASURING AND CHARACTERIZING WEB TRAFFIC.
9. Web Traffic Measurement.
Motivation for Web Measurement.
Motivation for Content Creators.
Motivation for Web Hosting Companies.
Motivation for Network Operators.
Motivation for Web/Networking Researchers.
Measurement Techniques.
Server Logging.
Proxy Logging.
Client Logging.
Packet Traces.
Active Measurement.
Proxy/Server Logs.
Common Log Format (CLF).
Extended Common Log Format (ECLF).
Preprocessing Measurement Data for Analysis.
Parsing Measurement Data.
Filtering Measurement Data.
Transforming Measurement Data.
Drawing Inferences From Measurement Data.
Limitations of HTTP Header Information.
Ambiguous Client/Server Identity.
Inferring User Actions.
Detecting Resource Modifications.
Case Studies.
Saskatchewan Server Log Study.
British Columbia Proxy Log Study.
Boston University Client Log Study.
AT&T Packet-Trace Study.
Summary.
10. Web Workload Characterization.
Workload Characterization.
Applications of Workload Models.
Selecting Workload Parameters.
Statistics and Probability Distributions.
Mean, Median, and Variance.
Probability Distributions.
HTTP Message Characteristics.
HTTP Request Methods.
HTTP Response Codes.
Web Resource Characteristics.
Content Types.
Resource Sizes.
Response Sizes.
Resource Popularity.
Resource Changes.
Temporal Locality.
Number of Embedded Resources.
User Behavior Characteristics.
Session and Request Arrivals.
Clicks per Session.
Request Interarrival Time.
Applying Workload Models.
Combining Workload Parameters.
Validating the Workload Model.
Generating Synthetic Traffic.
User Privacy.
Access to User-Level Data.
Information Available to Software Components.
Application of User-Level Data.
Summary.
V. WEB APPLICATIONS.
11. Web Caching.
The origins and goals of Web Caching.
Why cache?
What is cacheable?
Protocol-speci_c considerations.
Content-speci_c considerations.
Where is caching done?
How is caching done?
Deciding whether a message is cacheable.
Cache replacement and storing response in cache.
Returning a cached response.
Maintaining a cache.
Cache replacement.
Cache coherency.
Rate of change of resources.
Cache-related protocols.
Internet Cache Protocol (ICP).
Cache Array Resolution Protocol (CARP).
Cache Digest Protocol.
Web Cache Coordination Protocol (WCCP).
Cache software and hardware.
Cache software: The Squid cache.
Caching hardware.
Impediments to caching.
Cache busting.
Privacy issues in caching.
Caching versus replication.
Content distribution.
Content adaptation.
Summary.
12. Delivering Multimedia Streams.
Multimedia Streaming.
Audio and Video Data.
Multimedia Streaming Applications.
Properties of Multimedia Applications.
Delivering Multimedia Content.
Performance Requirements.
Limitations of IP Networks.
Multimedia-on-Demand Over HTTP.
Protocols for Multimedia Streaming.
Data Transport.
Session Establishment.
Session Description.
Presentation Description.
Real Time Streaming Protocol.
Similarities and Differences.
RTSP Request Methods.
RTSP Headers.
RTSP Status Codes.
Summary.
VI. Research Perspectives.
13. Research Perspectives in Caching.
Cache revalidation and invalidation.
Costs associated with revalidation.
Prevalidation.
The Piggybacking approach.
Server-driven invalidation.
End-to-end information exchange.
Server volumes.
Proxy filters.
Volumes and Filters: Practical details.
Volume Construction Algorithms.
Evaluation of volume construction algorithms.
End-to-end information exchange summary.
Prefetching.
DNS Prefetching.
Connection Prefetching.
HTTP Prefetching.
Trade-offs in Prefetching.
Summary.
14. Research Perspectives in Measurement.
Packet Monitoring of HTTP Traffic.
Tapping a Link.
Capturing Packets.
Demultiplexing Packets.
Reconstructing the Ordered Stream.
Extracting HTTP Messages.
Generating HTTP Traces.
Analyzing Web Server Logs.
Parsing and Filtering.
Transforming.
Publicly-Available Logs and Traces.
Measuring Multimedia Streams.
Static Analysis of Multimedia Resources.
Multimedia Server Logs.
Packet Monitoring of Multimedia Streams.
Multi-Layer Packet Monitoring.
Summary.
15. Research Perspectives in Protocol Issues.
Multiplexing HTTP Transfers.
WebMux: An experimental multiplexing Protocol.
TCP Control Block Interdependence.
Integrated Congestion Management.
Adding a Differencing Mechanism to HTTP/1.1.
Motivations for a differencing mechanism for HTTP messages.
Evaluation of delta algorithms.
Deployment issues of delta mechanism in HTTP/1.1.
Status of adding delta mechanism to HTTP/1.1.
HTTP/1.1 Protocol compliance.
Motivation for protocol compliance study.
Testing compliance of clients and proxies.
Methodology of testing compliance.
PROCOW: A large scale compliance study.
Summary of protocol compliance.
End-to-end measurements to study Web performance.
Identifying the factors in end-to-end performance.
Report on an end-to-end performance study.
Summary of end-to-end performance study.
Other extensions to HTTP.
Transparent Content Negotiation.
WebDAV|Web Distributed Authoring and Versioning.
An HTTP Extension Framework.
Summary.
Bibliography.
Index. 0201710889T04062001
1. Introduction.
Origin and Growth of the World Wide Web.
Historical Evolution of the Web.
The State of the Web.
Semantic Components of the Web.
Uniform Resource Identifer (URI).
HyperText Markup Language (HTML).
HyperText Transfer Protocol (HTTP).
Terms and Concepts.
Content on the Web.
Software Components.
Underlying Network.
Standardization.
Web.
Web Applications.
Topics Not Covered.
Tour Through the Book.
II. WEB SOFTWARE COMPONENTS.
2. Web Clients.
Client as a program.
Evolution of the browser.
Web-related browser functions.
Canonical Web transfer example.
Issuing a request from a browser.
Browser caching.
Request message headers.
Response handling.
Browser configuration.
Physical appearance.
Semantic choices.
Configuring browser for non-protocol functions.
Browser security issues.
Cookies.
Motivations for cookies.
Use of cookies in a browser.
User's control over cookies.
Privacy problems with cookies.
Spiders.
Searching on the Web.
Spider client.
Use of spiders in search engines.
Intelligent agents and special-purpose browsers.
Intelligent agents.
Special-purpose browsers.
Summary.
3. Web Proxies.
History and evolution of intermediaries.
High-level classification of proxies.
Caching proxy.
Transparent proxy.
Proxy applications.
Sharing access to the Web.
Caching responses.
Anonymizing clients.
Transforming requests and responses.
Gateway to non-HTTP systems.
Filtering requests and responses.
HTTP-related proxy roles.
Steps in request-response exchange with a proxy.
Handling HTTP requests and responses.
Proxy as Web server.
Proxy as Web client.
Example use of a proxy.
Proxy chaining and hierarchies.
Proxy configuration.
Proxy privacy issues.
Other kinds of proxies.
Reverse proxies or surrogates.
Interception proxies.
Summary.
4. Web Servers.
Web Site Versus Web Server.
Web Site.
Web Server.
Handling a Client Request.
Steps in Handling a Client Request.
Access Control.
Dynamically Generated Responses.
Creating and Using Cookies.
Sharing Information Across Requests.
Sharing HTTP Responses Across Requests.
Sharing Metadata Across Requests.
Server Architecture.
Event-Driven Server Architecture.
Process-Driven Server Architecture.
Hybrid Server Architecture.
Server Hosting.
Multiple Web Sites on a Single Machine.
Multiple Machines for a Single Web Site.
Case Study: Apache Web Server.
Resource Management.
HTTP Request Processing.
Summary.
III. WEB PROTOCOLS.
5. Protocols Underlying HTTP.
Internet Protocol.
Evolution of the Internet Architecture.
IP Design Goals.
IP Addresses.
IP Header Details.
Transmission Control Protocol 0.
Socket Abstraction 1.
Ordered Reliable Byte Stream.
Opening and Closing a TCP Connection.
Sliding-Window Flow Control.
Retransmission of Lost Packets.
TCP Congestion Control.
TCP Header Details.
Domain Name System.
DNS Resolver.
DNS Architecture.
DNS Protocol.
DNS Queries and the Web.
DNS-Based Web Server Load Balancing.
Application-Layer Protocols.
Telnet Protocol.
File Transfer Protocol.
Simple Mail Transfer Protocol.
The Network News Transfer Protocol.
Properties of Application-Layer Protocols.
Summary.
6. HTTP Protocol Design and Description.
Overview of HTTP.
Protocol properties.
Protocol infuences.
HTTP language elements.
HTTP terms.
HTTP/1.0 Request Methods.
HTTP/1.0 Headers.
HTTP/1.0 Response classes.
HTTP Extensibility.
SSL and Security.
SSL.
HTTPS: Using SSL in Web exchanges.
Security in HTTP/1.0.
Protocol compliance and interoperability.
Version number and interoperability.
MUST, SHOULD, MAY requirement levels.
Summary.
7. HTTP/1.1 221.
The evolution of HTTP/1.1 protocol.
History of evolution.
Problems with HTTP/1.0 25.
New concepts in HTTP/1.1.
Methods, headers, response codes in 1.0 and 1.1.
Old and new request methods.
Old and new headers.
Old and new response codes.
Caching.
Caching related terms.
Caching in HTTP/1.0.
Caching in HTTP/1.1.
Bandwidth optimization.
The Range request.
The Expect/Continue mechanism.
Compression.
Connection Management.
The Connection: Keep-Alive mechanism of HTTP/1.0.
Evolution of HTTP/1.1 persistent connection mechanism.
The Connection header.
Pipelining on persistent connections.
Closing persistent connections.
Message transmission.
Extensibility.
Learning about the server.
Learning about intermediate servers.
Upgrading to other protocols.
Internet address conservation.
Content negotiation.
Security, authentication, and integrity.
Security and authentication.
Integrity.
The role of proxies in HTTP/1.1.
Types of proxies.
Syntactic requirements on an HTTP/1.1 proxy.
Semantic requirements on an HTTP/1.1 proxy.
Other miscellaneous changes.
Method-related miscellaneous changes.
Header-related miscellaneous changes.
Response-code-related miscellaneous changes.
Summary.
8. HTTP/TCP Interaction.
TCP Timers.
Retransmission Timer.
Slow-Start Restart.
The TIME WAIT State.
HTTP/TCP Layering.
Aborted HTTP Transfers.
Nagle's Algorithm.
Delayed Acknowledgments.
Multiplexing TCP Connections.
Motivation for Parallel Connections.
Problems with Parallel Connections.
Server Overheads.
Combining System Calls.
Managing Multiple Connections.
Summary.
IV. MEASURING AND CHARACTERIZING WEB TRAFFIC.
9. Web Traffic Measurement.
Motivation for Web Measurement.
Motivation for Content Creators.
Motivation for Web Hosting Companies.
Motivation for Network Operators.
Motivation for Web/Networking Researchers.
Measurement Techniques.
Server Logging.
Proxy Logging.
Client Logging.
Packet Traces.
Active Measurement.
Proxy/Server Logs.
Common Log Format (CLF).
Extended Common Log Format (ECLF).
Preprocessing Measurement Data for Analysis.
Parsing Measurement Data.
Filtering Measurement Data.
Transforming Measurement Data.
Drawing Inferences From Measurement Data.
Limitations of HTTP Header Information.
Ambiguous Client/Server Identity.
Inferring User Actions.
Detecting Resource Modifications.
Case Studies.
Saskatchewan Server Log Study.
British Columbia Proxy Log Study.
Boston University Client Log Study.
AT&T Packet-Trace Study.
Summary.
10. Web Workload Characterization.
Workload Characterization.
Applications of Workload Models.
Selecting Workload Parameters.
Statistics and Probability Distributions.
Mean, Median, and Variance.
Probability Distributions.
HTTP Message Characteristics.
HTTP Request Methods.
HTTP Response Codes.
Web Resource Characteristics.
Content Types.
Resource Sizes.
Response Sizes.
Resource Popularity.
Resource Changes.
Temporal Locality.
Number of Embedded Resources.
User Behavior Characteristics.
Session and Request Arrivals.
Clicks per Session.
Request Interarrival Time.
Applying Workload Models.
Combining Workload Parameters.
Validating the Workload Model.
Generating Synthetic Traffic.
User Privacy.
Access to User-Level Data.
Information Available to Software Components.
Application of User-Level Data.
Summary.
V. WEB APPLICATIONS.
11. Web Caching.
The origins and goals of Web Caching.
Why cache?
What is cacheable?
Protocol-speci_c considerations.
Content-speci_c considerations.
Where is caching done?
How is caching done?
Deciding whether a message is cacheable.
Cache replacement and storing response in cache.
Returning a cached response.
Maintaining a cache.
Cache replacement.
Cache coherency.
Rate of change of resources.
Cache-related protocols.
Internet Cache Protocol (ICP).
Cache Array Resolution Protocol (CARP).
Cache Digest Protocol.
Web Cache Coordination Protocol (WCCP).
Cache software and hardware.
Cache software: The Squid cache.
Caching hardware.
Impediments to caching.
Cache busting.
Privacy issues in caching.
Caching versus replication.
Content distribution.
Content adaptation.
Summary.
12. Delivering Multimedia Streams.
Multimedia Streaming.
Audio and Video Data.
Multimedia Streaming Applications.
Properties of Multimedia Applications.
Delivering Multimedia Content.
Performance Requirements.
Limitations of IP Networks.
Multimedia-on-Demand Over HTTP.
Protocols for Multimedia Streaming.
Data Transport.
Session Establishment.
Session Description.
Presentation Description.
Real Time Streaming Protocol.
Similarities and Differences.
RTSP Request Methods.
RTSP Headers.
RTSP Status Codes.
Summary.
VI. Research Perspectives.
13. Research Perspectives in Caching.
Cache revalidation and invalidation.
Costs associated with revalidation.
Prevalidation.
The Piggybacking approach.
Server-driven invalidation.
End-to-end information exchange.
Server volumes.
Proxy filters.
Volumes and Filters: Practical details.
Volume Construction Algorithms.
Evaluation of volume construction algorithms.
End-to-end information exchange summary.
Prefetching.
DNS Prefetching.
Connection Prefetching.
HTTP Prefetching.
Trade-offs in Prefetching.
Summary.
14. Research Perspectives in Measurement.
Packet Monitoring of HTTP Traffic.
Tapping a Link.
Capturing Packets.
Demultiplexing Packets.
Reconstructing the Ordered Stream.
Extracting HTTP Messages.
Generating HTTP Traces.
Analyzing Web Server Logs.
Parsing and Filtering.
Transforming.
Publicly-Available Logs and Traces.
Measuring Multimedia Streams.
Static Analysis of Multimedia Resources.
Multimedia Server Logs.
Packet Monitoring of Multimedia Streams.
Multi-Layer Packet Monitoring.
Summary.
15. Research Perspectives in Protocol Issues.
Multiplexing HTTP Transfers.
WebMux: An experimental multiplexing Protocol.
TCP Control Block Interdependence.
Integrated Congestion Management.
Adding a Differencing Mechanism to HTTP/1.1.
Motivations for a differencing mechanism for HTTP messages.
Evaluation of delta algorithms.
Deployment issues of delta mechanism in HTTP/1.1.
Status of adding delta mechanism to HTTP/1.1.
HTTP/1.1 Protocol compliance.
Motivation for protocol compliance study.
Testing compliance of clients and proxies.
Methodology of testing compliance.
PROCOW: A large scale compliance study.
Summary of protocol compliance.
End-to-end measurements to study Web performance.
Identifying the factors in end-to-end performance.
Report on an end-to-end performance study.
Summary of end-to-end performance study.
Other extensions to HTTP.
Transparent Content Negotiation.
WebDAV|Web Distributed Authoring and Versioning.
An HTTP Extension Framework.
Summary.
Bibliography.
Index. 0201710889T04062001