HTTP

Internet Engineering

Introduction

• HTTP is the transfer protocol for web applications
• H T T P: Hyper Text Transfer Protocol
• HTTP 1.0 (RFC 1945), HTTP 1.1 (RFC 2068), HTTP 2 (RFC 7540)
• In fact, it can be used to transfer everything (not only hyper text)
• Text Documents e.g. HTML, XML, JSON, etc.
• Multimedia e.g. JGP, GIF, MP4, MKV, etc.
• Application e.g. PDF, ZIP, etc.

Introduction (Cont.)

• HTTP uses the client/server paradigm
• HTTP server provides resource
• HTTP client (usually web browser) gets resource
• But not pure client/server communication
• Proxies
• Caches
• ...

Introduction (Cont.)

• HTTP is an application layer protocol
• HTTP assumes reliable communication
• over TCP
• default (server) port: 80
• client port is chosen randomly per connection
• HTTP is Stateless
• Server does not keep history/state of client
• High performance & Low Complexity
• Problematic in some applications (sessions)
• Cookies
• JSON Web Tokens

Data Resources to Transfer

• HTTP is the protocol to transfer data between server and client (usually from server to client)
• Which data?
• It can be anything
• In web, usually, it is a resource/object on server
• Each resource must be identified/located uniquely
• URI (Uniform Resource Identifier)
• URL (Uniform Resource Locator)
• URIs identify and URLs locate; however, locators are also identifiers, so

every URL is also a URI, but there are URIs which are not URLs.

URI in Action

• A Uniform Resource Name (URN) is a URI that identifies a resource by name in a particular namespace.
• International Standard Book Number (ISBN) system

URL

• https://sbu.ac.ir/SitePages/Home.aspx
• https://www.bing.com/search?q=Hello+World&form=QBLH&sp=-1&pq=&sc=0-0&qs=n&sk=&cvid=7F2B496642F94D5F95989756B4FF60EE
• file:///home/parham/Documents/Git/parham/dotfiles
• http://libgen.is
• ftp://speedtest.tele2.net

URL (Cont.)

• Scheme: The application layer protocol
• HTTP: The web protocol
• HTTPS: Secure HTTP
• FTP: File Transfer Protocol
• File: Access to a local file
• javascript: Run javascript code
• mailto: Send mail to given address
• etc.

URL (Cont.)

• Path: The path of the object on host filesystem
• E.g. web server root directory is /var/www/
• http://www.example.com/1.html → /var/www/1.html
• http://www.example.com/1/2/3.jpg → /var/www/1/2/3.jpg
• http://www.example.com/1/2/../3.jpg → /var/www/1/3.jpg 🤨

URL (Cont.)

• Query: A mechanism to pass information from client to active pages or forms
• Fill information in a university registration form
• Ask Bing! to search a phrase
• Starts with "?"
• name=value format
• "&" is the border between multiple parameters
• Try this out, by clicking me

URL (Cont.)

• Frag: A name for a part of resource
• A section in a document
• Handled by browser
• Browser gets whole resource (doc) from server
• In display time, it jumps to the specific part
• Try this out, by clicking me

URL (Cont.)

• Domain names are case insensitive according to RFC 4343.
• The rest of URL is sent to the server via the GET method and etc. This may be case sensitive or not.

URL (Cont.)

• URL is encoded by client before transmission
• How: Each byte is divided into two 4-bit group, hexadecimal of the 4-bits are prefixed by %
• ~ → 126 → %7E
• What & Why?
• Non-ASCII (e.g., Persian Characters, Emoji)
• Reserved character when are not used for special role
• Unsafe character, e.g. space, %, ...

https://ganj.irandoc.ac.ir/api/v1/search/main?keywords=hellow%20world

URL in Action (Cont.)

2950	113.658623720	192.168.73.191	3.220.112.94	TCP	74	59890 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=335902154 TSecr=0 WS=128
2956	113.848038152	3.220.112.94	192.168.73.191	TCP	74	80 → 59890 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=1460 SACK_PERM=1 TSval=1360379473 TSecr=335902154 WS=256
2957	113.848146169	192.168.73.191	3.220.112.94	TCP	66	59890 → 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=335902344 TSecr=1360379473
2958	113.848626070	192.168.73.191	3.220.112.94	HTTP	503	GET /bytes/1378 HTTP/1.1
2962	114.079894429	3.220.112.94	192.168.73.191	TCP	66	80 → 59890 [ACK] Seq=1 Ack=438 Win=28160 Len=0 TSval=1360379525 TSecr=335902344
2963	114.080912334	3.220.112.94	192.168.73.191	HTTP	1683	HTTP/1.1 200 OK
2964	114.080950090	192.168.73.191	3.220.112.94	TCP	66	59890 → 80 [ACK] Seq=438 Ack=1618 Win=62720 Len=0 TSval=335902576 TSecr=1360379526
  

How does HTTP work? Transactions

• HTTP data transfer is a collection of transactions
• Each transaction is composed of 2 HTTP messages
• Requests are identified by methods
• Method: The action that client asks from server
• Responses are identified by status codes
• Status: The result of the requested action

HTTP Transactions

HTTP Transactions in Web

• (Typically) each web page contains multiple resources
• The main skeleton HTML page
• Some linked materials: figures, videos, JS, CSS, etc.
• Displaying a web page by a browser
• Get the HTML page (first transaction)
• Try to display the page (rendering)
• Other resources are linked to the page
• Get the resources (subsequent transactions)

HTTP Transactions in Web (Cont.)

• HTTP Transactions & TCP Connections
1. Non-persistent
• A new TCP connection per object
• Network overhead + Connection establish delay + Resource intensive
• Parallel connections speed up browsing
2. Persistent
• Get multiple objects using a single TCP connection
• No extra processing & networking overhead
• Poor performance if implemented in serial manner
• Pipeline requests speed up browsing (HTTP/1.1)

HTTP/2 for a Faster Web

HTTP Pipelining

• HTTP Pipelining is not difficult to deploy, it is impossible.
• it still allowed a single large or slow response to block all others that followed.

HTTP/2 Multiplexing

• Multiplexing allows multiple request-response messages to be in flight over a single HTTP/2 connection, at the same time.

HTTP/2 for a Faster Web (Cont.)

HTTP Messages

• HTTP is text-based protocol
• Human readable headers
• The header is composed of some lines

HTTP Methods (Cont.)

• GET: Retrieve resource from server
• HEAD: Similar to GET but the resource itself is not retrieved, just the HTTP response header
• Useful for debugging or some other applications
• POST: Submit data to be processed by the specified resource
• Data itself is enveloped in message body

HTTP Methods (Cont.)

• DELETE: Remove the resource
• Not popular in web, can be used in other applications
• PUT: Add message body as the specified resource to server
• PATCH: A PATCH request is considered a set of instructions on how to modify a resource. Contrast this with PUT; which is a complete representation of a resource.
• TRACE: Server echoes back the received message
• For troubleshooting & debugging
• OPTIONS: Request the list of supported methods by server on the resource

HTTP Responses

• The message for the result/response of the requested action
• Which responses?
• Basic responses
• Success
• Failure
• Bad client request
• Server problem
• ...
• Others
• E.g., Redirection to other resources

HTTP Responses (Cont.)

• 2xx
• Successful responses
• 200: OK
• 201: Created
• 204: No Content
• 4xx
• Client errors
• 400: Bad Request
• 401: Unauthorized (Authorization required)
• 403: Forbidden
• 404: Not Found
• 405: Method Not Allowed

HTTP Responses (Cont.)

• 5xx
• Server errors
• 500: Internal Server Error
• 501: Not Implemented
• 503: Service Unavailable
• 3xx
• Redirects
• 301: Moved Permanently
• 302: Found
• redirect status response code indicates that the resource requested has been temporarily moved to the URL given by the Location header
• 307: Moved Temporarily
• Resource has been moved, Redirection
• Location header contains the new location of resource
• 304: Not Modified

HTTP Responses (Cont.)

• 1xx
• Informational responses
• 101: Switching Protocol
• This code is sent in response to an Upgrade request header from the client, and indicates the protocol the server is switching to.

HTTP Headers

• Headers are additional information that is sent by client to server and vice versa
• Most (almost all) are optional
• Which headers?
• Information about client
• Information about server
• Information about the requested resource
• Information about the response
• Security/Authentication
• ...

HTTP Headers

• General headers
• Appear both on request & response messages
• Request headers
• Information about request
• Response headers
• Information about response
• Entity headers
• Information about body (size, ...)
• Extension headers
• New headers (not standard)

General Headers

• Date: Date & Time that message is created
• Connection: Close or Keep-Alive
• Close: Non-persistent connection
• Keep-Alive: Persistent connection
• Via: Information about the intermediate nodes between two sides
• Proxy servers

Request Headers

• Host: The name of the server (required, why?)
• Referer: URL that contains requested URL
• Information about the client
• User-Agent: The client program
• Accept: The acceptable media types
• Accept-Encoding: The acceptable encoding
• Accept-Language: The acceptable language

Request Headers (Cont.)

• Range: Specific range (in byte) of resource
• Authorization: Response to the authenticate
• Will be discussed later
• Cookie: To return back the cookies
• Will be discussed later
• If-Modified-Since: Request is processed if the objected is modified since the specified time.
• Used in Web Caching
• Will be discussed later

Response Header

• Server: Information about server
• WWW-Authenticate: Used to specify authentication parameters by server
• Proxy-Authenticate: Used to specify authentication parameters by proxy
• Set-Cookie: To send a cookie to client
• Location: The location of entity to redirect client

Response Header (Cont.)

• Last-Modified: The date and time of last modification of entity
• Content-Range: Range of this entity in the entire resource
• Expires: The date and time at which the entity will expire

Extension Headers

• Custom proprietary headers have historically been used with an X- prefix, but this convention was deprecated in June 2012
• implementation-specific and private-use parameters could at least incorporate the organization's name
• ExampleInc-foo
• VND.ExampleInc.foo (vnd stands for vendor)
• or primary domain name
• com.example.foo
• http://example.com/foo

HTTP Tools

• cURL
• Postman
• Hoppscotch
• HTTPie

Stateless Problem

• HTTP is a stateless protocol
• Server does not remember it's client
• How to personalize pages (personal portal)?
• Use http header: X-Forwarded-For
• Is not usually sent by browsers
• Find client IP address from TCP connection
• The problem is NAT
• Clients move but IP address does not

Cookies (Cont.)

• Limitation
• Cannot be used to store large data
• At least 300 cookies
• At least 4096 bytes per cookie
• At least 20 cookies per unique host or domain name
• Cookies are text files
• No virus spread
• There is not any request from server to read cookies
• By default cookies are sent by browser
• Browser checks URL and finds appropriate cookies

Cookies (Cont.)

• Client can control cookies
• Disable cookies: no cookie is saved & used
• View & Delete cookies
• Server can control cookies by its attributes
• Expiration time
• Domain
• Path
• Security
• ...

Cookies Attributes: Domain & Path

• Validity check by major browsers
• Domain names must start with dot
• Some browsers accept names without dot as domain
• Contrary to earlier specifications, leading dots in domain names (.example.com) are ignored.
• Don’t accept for other domains than the base domain
• Don’t accept cookies for sub-domains
• Accept cookies for higher domains
• Except the top level domains, e.g., .com, .ac.ir
• Accept cookies for other (sub or higher) paths
• A path that must exist in the requested URL, or the browser won't send the Cookie header.

Single Sign-on (SSO)

• e.g. CAS Enterprise Single Sign-On

HTTP Proxy Applications

• Authentication
• Client side: Authenticate clients before they access web
• Server side: Authenticate clients before access the server
• Accounting: Log client activities
• Security: Analyze request before sending it to server
• Integrated in modern firewalls
• Filtering: Limit access to specified contents
• Anonymizer: Anonymous web browsing
• Caching (more details in the following slide)

Authentication vs Authorization

• Authentication is the verification of the credentials of the connection attempt.
• This process consists of sending the credentials from the remote access client to the remote access server in an either plaintext or encrypted form by using an authentication protocol.
• Authentication is the verification that the connection attempt is allowed.
• Authorization occurs after successful authentication.

HTTP Authentication

• All resources are not public in web; e.g.,
• Financial documents, Customer information, ...
• HTTP has two (similar) authentications
• Basic: Base64 encoded user:pass [The username itself cannot contain a colon]
• Digest: Plain username + Digest of pass
• Steps are the same
1. Client-side app (browser) request resource from server
2. Server refuses with 401 Unauthorized
3. Client-side app ask Username & Password from user
4. Client send Username & Password to server
5. Server authenticates and allows.
• Authentication information are sent by every request until end of current session

Base64

• Base64 is a group of binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation.

HTTP Authentication (Cont.)

HTTP Authentication (Cont.)

GET /basic-auth/admin/admin HTTP/1.1
Host: httpbin.org
Authorization: Basic YWRtaW46YWRtaW4=

  

HTTP/1.1 200 OK
Date: Mon, 07 Sep 2020 14:14:25 GMT
Content-Type: application/json
Content-Length: 48
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "authenticated": true,
  "user": "admin"
}
  

Digest Authentication

• Basic authentication is insecure
• Password is sent in base64 encoding
• Attacker can easily find it
• Digest authentication: Don’t send password
• Send its digest (hash)
• Digest/hash function
• One way function, irreversible
• Attacker cannot find password 😌
• But! Reply attack 😞
• Attacker resends the same digest then he will be authenticated
• Use Nonce
• Digest authentication uses nonce and digest together

Bearer Authentication

• Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens
• The name Bearer authentication can be understood as give access to the bearer of this token.

References 📚

• Prof. Bahador Bakhshi's Internet Eng. Course's Slides
• HTTP: Learn your browser's language!
• What is the difference between a URI, a URL and a URN?
• HTTP/2 for a Faster Web
• REST: Good Practices for API Design
• What is the maximum size of a cookie, and how many can be stored in a browser for each web site?
• Base64
• Digest access authentication