2019-11-11

源码学习-net/http

本文总阅读量次

package net/http是Go语言Web应用的基础，学习net/http源码不仅可以学习网络编程中常见范式，也是学习Golang接口设计哲学。通过http包提供了HTTP客户端和服务端的实现。同时使用这个包能很简单地对web的路由，静态文件，模版，cookie等数据进行设置和操作。

主要涉及以下源码文件：

net/net.go
net/server.go
net/http.go
net/transfer.go
sync/pool.go
sync/mutex.go

HTTP

HTTP协议

在HTTP中，客户端总是通过建立一个连接与发送一个HTTP请求来发起一个事务
HTTP协议是无状态的，同一个客户端的这次请求和上次请求是没有对应关系的，对HTTP服务器来说，它并不知道这两个请求是否来自同一个客户端。为了解决这个问题， Web程序引入了Cookie机制来维护连接的可持续状态。

HTTP请求包（浏览器信息）

Request

Request包分为3部分，第一部分叫Request line（请求行）, 第二部分叫Request header（请求头）,第三部分是body（主体）。header和body之间有个空行，请求包的例子所示:

GET /domains/example/ HTTP/1.1		//请求行: 请求方法 请求URI HTTP协议/协议版本
Host：www.iana.org				//服务端的主机名
User-Agent：Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4			//浏览器信息
Accept：text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8	//客户端能接收的MIME
Accept-Encoding：gzip,deflate,sdch		//是否支持流压缩
Accept-Charset：UTF-8,*;q=0.5		//客户端字符编码集
//空行,用于分割请求头和消息体
//消息体,请求资源参数,例如POST传递的参数

HTTP协议定义了很多与服务器交互的请求方法，最基本的有4种，分别是GET,POST,PUT,DELETE。一个URL地址用于描述一个网络上的资源，而HTTP中的GET, POST, PUT, DELETE就对应着对这个资源的查，增，改，删4个操作。我们最常见的就是GET和POST了。GET一般用于获取/查询资源信息，而POST一般用于更新资源信息。

HTTP响应包

Respons

HTTP/1.1 200 OK						//状态行
Server: nginx/1.0.8					//服务器使用的WEB软件名及版本
Date:Date: Tue, 30 Oct 2012 04:14:25 GMT		//发送时间
Content-Type: text/html				//服务器发送信息的类型
Transfer-Encoding: chunked			//表示发送HTTP包是分段发的
Connection: keep-alive				//保持连接状态
Content-Length: 90					//主体内容长度
//空行 用来分割消息头和主体
!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"... //消息体

HTTP网络应用

理解 HTTP 构建的网络应用只要关注两个端—客户端（clinet）和服务端（server），两个端的交互来自 clinet 的 request，以及server端的response。所谓的http服务器，主要在于如何接受 clinet 的 request，并向client返回response。

接收request的过程中，最重要的莫过于路由（router），即实现一个Multiplexer器。Go中既可以使用内置的mutilplexer — DefautServeMux，也可以自定义。Multiplexer路由的目的就是为了找到处理器函数（handler），后者将对request进行处理，同时构建response。

简单总结就是这个流程为：

1	Clinet -> Requests -> [Multiplexer(router) -> handler -> Response -> Clinet

Golang中的Multiplexer基于ServeMux结构，同时也实现了Handler接口

hander函数：具有func(w http.ResponseWriter, r *http.Requests)签名的函数
handler处理器(函数): 经过HandlerFunc结构包装的handler函数，它实现了ServeHTTP接口方法的函数。调用handler处理器的ServeHTTP方法时，即调用handler函数本身。
handler对象：实现了Handler接口ServeHTTP方法的结构。

一个简单的web服务器

import (
    "io"
    "log"
    "net/http"
)
 
func GoServer(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "Hello, this is a GoServer")
}
 
func main() {
    http.HandleFunc("/", GoServer)
    err := http.ListenAndServe(":8090", nil)
    if err != nil {
        log.Fatal("ListenAndServer ", err)
    }
}

注册路由

http.HandleFunc("/", GoServer)即是在注册路由
函数：

1
2
3

func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
    DefaultServeMux.HandleFunc(pattern, handler)
}

可以看出http.HandleFunc选取了DefaultServeMux作为multiplexer

ServeMux

type ServeMux struct {
    mu    sync.RWMutex
    m     map[string]muxEntry
    hosts bool 
}

type muxEntry struct {
    explicit bool
    h        Handler
    pattern  string
}

m:一个map，key是一些url模式，value是一个muxEntry结构，muxEntry结构里存储了具体的url模式和handler

// Handle registers the handler for the given pattern.
// If a handler already exists for pattern, Handle panics.
func (mux *ServeMux) Handle(pattern string, handler Handler) {
    mux.mu.Lock()
    defer mux.mu.Unlock()

    if pattern == "" {
        panic("http: invalid pattern " + pattern)
    }
    if handler == nil {
        panic("http: nil handler")
    }
    if mux.m[pattern].explicit {
        panic("http: multiple registrations for " + pattern)
    }

    if mux.m == nil {
        mux.m = make(map[string]muxEntry)
    }
    mux.m[pattern] = muxEntry{explicit: true, h: handler, pattern: pattern}

    if pattern[0] != '/' {
        mux.hosts = true
    }

    // Helpful behavior:
    // If pattern is /tree/, insert an implicit permanent redirect for /tree.
    // It can be overridden by an explicit registration.
    n := len(pattern)
    if n > 0 && pattern[n-1] == '/' && !mux.m[pattern[0:n-1]].explicit {
        // If pattern contains a host name, strip it and use remaining
        // path for redirect.
        path := pattern
        if pattern[0] != '/' {
            // In pattern, at least the last character is a '/', so
            // strings.Index can't be -1.
            path = pattern[strings.Index(pattern, "/"):]
        }
        url := &url.URL{Path: path}
        mux.m[pattern[0:n-1]] = muxEntry{h: RedirectHandler(url.String(), StatusMovedPermanently), pattern: pattern}
    }
}

ServeMux的Handle方法将会对pattern和handler函数做一个map映射，把handler和pattern模式绑定到map[string]muxEntry的map上，其中muxEntry保存了更多pattern和handler的信息

handler

1
2
3

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

go的http服务都是基于handler进行处理，ServeMux使用handler并调用其ServeHTTP方法处理请求并返回相应

DefaultServeMux

// NewServeMux allocates and returns a new ServeMux.
func NewServeMux() *ServeMux { return new(ServeMux) }

// DefaultServeMux is the default ServeMux used by Serve.
var DefaultServeMux = &defaultServeMux

var defaultServeMux ServeMux

DefaultServeMux是ServeMux的一个实例，默认创建DefaultServeMux，也可以自己 NewServeMux创建

函数调用关系

1
2
3

http.HandleFunc-->  func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request))

-->func (mux *ServeMux) Handle(pattern string, handler Handler)

开始监听

http.ListenAndServe(":8090")即是在开启监听
函数：

func ListenAndServe(addr string, handler Handler) error {
    server := &Server{Addr: addr, Handler: handler}
    return server.ListenAndServe()
}

利用参数addr和handler创建一个Server类型的变量server，然后调用这个结构体里的成员函数ListenAndServe()

Server

type Server struct {
    Addr         string        
    Handler      Handler       
    ReadTimeout  time.Duration 
    WriteTimeout time.Duration 
    TLSConfig    *tls.Config   

    MaxHeaderBytes int

    TLSNextProto map[string]func(*Server, *tls.Conn, Handler)

    ConnState func(net.Conn, ConnState)
    ErrorLog *log.Logger
    disableKeepAlives int32     nextProtoOnce     sync.Once 
    nextProtoErr      error     
}

Handler:保留Handler接口,如果Server结构没有提供Handler结构对象，那么会使用DefaultServerMux做multiplexer

ListenAndServe

func (srv *Server) ListenAndServe() error {
    addr := srv.Addr
    if addr == "" {
        addr = ":http"
    }
    ln, err := net.Listen("tcp", addr)
    if err != nil {
        return err
    }
    return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}

监听srv.Addr指定的TCP网络，并且调用srv.Serve()函数来处理收到的请求,最后将监听的TCP对象传入Serve方法

srv.Serve

func (srv *Server) Serve(l net.Listener) error {
    defer l.Close()
    if fn := testHookServerServe; fn != nil {
        fn(srv, l)
    }
    var tempDelay time.Duration // how long to sleep on accept failure

    if err := srv.setupHTTP2_Serve(); err != nil {
        return err
    }

    // TODO: allow changing base context? can't imagine concrete
    // use cases yet.
    baseCtx := context.Background()
    ctx := context.WithValue(baseCtx, ServerContextKey, srv)
    ctx = context.WithValue(ctx, LocalAddrContextKey, l.Addr())
    for {
        rw, e := l.Accept()
        if e != nil {
            if ne, ok := e.(net.Error); ok && ne.Temporary() {
                if tempDelay == 0 {
                    tempDelay = 5 * time.Millisecond
                } else {
                    tempDelay *= 2
                }
                if max := 1 * time.Second; tempDelay > max {
                    tempDelay = max
                }
                srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)
                time.Sleep(tempDelay)
                continue
            }
            return e
        }
        tempDelay = 0
        c := srv.newConn(rw)
        c.setState(c.rwc, StateNew) // before Serve can return
        go c.serve(ctx)
    }
}

context:一个map，设置一些背景变量，使用方法是：context.WithValue(parentCtx,key,value)
for循环:l.Accept()阻塞直到获取到一个net.Conn，之后通过srv.newConn(rw)建立一个server.conn(属于私有变量，不对外暴露)，并设置状态为StateNew
启动goroutine处理连接：调用go c.serve(ctx)。

server.conn

// A conn represents the server side of an HTTP connection.
type conn struct {
    // server is the server on which the connection arrived.
    // Immutable; never nil.
    server *Server

    // rwc is the underlying network connection.
    // This is never wrapped by other types and is the value given out
    // to CloseNotifier callers. It is usually of type *net.TCPConn or
    // *tls.Conn.
    rwc net.Conn

    // remoteAddr is rwc.RemoteAddr().String(). It is not populated synchronously
    // inside the Listener's Accept goroutine, as some implementations block.
    // It is populated immediately inside the (*conn).serve goroutine.
    // This is the value of a Handler's (*Request).RemoteAddr.
    remoteAddr string

    // tlsState is the TLS connection state when using TLS.
    // nil means not TLS.
    tlsState *tls.ConnectionState

    // werr is set to the first write error to rwc.
    // It is set via checkConnErrorWriter{w}, where bufw writes.
    werr error

    // r is bufr's read source. It's a wrapper around rwc that provides
    // io.LimitedReader-style limiting (while reading request headers)
    // and functionality to support CloseNotifier. See *connReader docs.
    r *connReader

    // bufr reads from r.
    // Users of bufr must hold mu.
    bufr *bufio.Reader

    // bufw writes to checkConnErrorWriter{c}, which populates werr on error.
    bufw *bufio.Writer

    // lastMethod is the method of the most recent request
    // on this connection, if any.
    lastMethod string

    // mu guards hijackedv, use of bufr, (*response).closeNotifyCh.
    mu sync.Mutex

    // hijackedv is whether this connection has been hijacked
    // by a Handler with the Hijacker interface.
    // It is guarded by mu.
    hijackedv bool
}

mu类型是sync.Mutex,mu就是持有对象锁的那个实例。可以看到conn的hijackedv属性就是通过mu来进行维护的，目的是防止同步更新问题。

处理请求

conn.serve

// Serve a new connection.
func (c *conn) serve(ctx context.Context) {
    c.remoteAddr = c.rwc.RemoteAddr().String()
    defer func() {
        if err := recover(); err != nil {
            const size = 64 << 10
            buf := make([]byte, size)
            buf = buf[:runtime.Stack(buf, false)]
            c.server.logf("http: panic serving %v: %v\n%s", c.remoteAddr, err, buf)
        }
        if !c.hijacked() {
            c.close()
            c.setState(c.rwc, StateClosed)
        }
    }()

    if tlsConn, ok := c.rwc.(*tls.Conn); ok {
        if d := c.server.ReadTimeout; d != 0 {
            c.rwc.SetReadDeadline(time.Now().Add(d))
        }
        if d := c.server.WriteTimeout; d != 0 {
            c.rwc.SetWriteDeadline(time.Now().Add(d))
        }
        if err := tlsConn.Handshake(); err != nil {
            c.server.logf("http: TLS handshake error from %s: %v", c.rwc.RemoteAddr(), err)
            return
        }
        c.tlsState = new(tls.ConnectionState)
        *c.tlsState = tlsConn.ConnectionState()
        if proto := c.tlsState.NegotiatedProtocol; validNPN(proto) {
            if fn := c.server.TLSNextProto[proto]; fn != nil {
                h := initNPNRequest{tlsConn, serverHandler{c.server}}
                fn(c.server, tlsConn, h)
            }
            return
        }
    }

    // HTTP/1.x from here on.

    c.r = &connReader{r: c.rwc}
    c.bufr = newBufioReader(c.r)
    c.bufw = newBufioWriterSize(checkConnErrorWriter{c}, 4<<10)

    ctx, cancelCtx := context.WithCancel(ctx)
    defer cancelCtx()

    for {
        w, err := c.readRequest(ctx)
        if c.r.remain != c.server.initialReadLimitSize() {
            // If we read any bytes off the wire, we're active.
            c.setState(c.rwc, StateActive)
        }
        if err != nil {
            if err == errTooLarge {
                // Their HTTP client may or may not be
                // able to read this if we're
                // responding to them and hanging up
                // while they're still writing their
                // request. Undefined behavior.
                io.WriteString(c.rwc, "HTTP/1.1 431 Request Header Fields Too Large\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\n431 Request Header Fields Too Large")
                c.closeWriteAndWait()
                return
            }
            if err == io.EOF {
                return // don't reply
            }
            if neterr, ok := err.(net.Error); ok && neterr.Timeout() {
                return // don't reply
            }
            var publicErr string
            if v, ok := err.(badRequestError); ok {
                publicErr = ": " + string(v)
            }
            io.WriteString(c.rwc, "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\n400 Bad Request"+publicErr)
            return
        }

        // Expect 100 Continue support
        req := w.req
        if req.expectsContinue() {
            if req.ProtoAtLeast(1, 1) && req.ContentLength != 0 {
                // Wrap the Body reader with one that replies on the connection
                req.Body = &expectContinueReader{readCloser: req.Body, resp: w}
            }
        } else if req.Header.get("Expect") != "" {
            w.sendExpectationFailed()
            return
        }

        // HTTP cannot have multiple simultaneous active requests.[*]
        // Until the server replies to this request, it can't read another,
        // so we might as well run the handler in this goroutine.
        // [*] Not strictly true: HTTP pipelining. We could let them all process
        // in parallel even if their responses need to be serialized.
        serverHandler{c.server}.ServeHTTP(w, w.req)
        w.cancelCtx()
        if c.hijacked() {
            return
        }
        w.finishRequest()
        if !w.shouldReuseConnection() {
            if w.requestBodyLimitHit || w.closedRequestBodyEarly() {
                c.closeWriteAndWait()
            }
            return
        }
        c.setState(c.rwc, StateIdle)
    }
}

初始化bufr和bufw
从底层读取客户端发送的数据
调用用户的handler：serverHandler{c.server}.ServeHTTP(w, w.req)，这句实际上调用的是c.server.Handler.ServeHTTP()。Handler就是DefaultServeMux或者用户指定的ServeMux，根据用户定义路由规则，来具体调用用户的业务逻辑方法。
处理工作：异常处理，资源回收，状态更新。
参考链接：http/net的源码解读