erikdubbelboer

erikdubbelboer

I don't fork unless I make a pull request.

Member Since 10 years ago

Amsterdam

Experience Points
241
follower
Lessons Completed
6
follow
Best Reply Awards
127
repos
erikdubbelboer Most Used Languages
erikdubbelboer GitHub Stats
Jun
22
1 day ago
Activity icon
issue

erikdubbelboer issue comment valyala/fasthttp

erikdubbelboer
erikdubbelboer

Error when sending more than 100 requests to a host via PipelineClient .

Hello :) ,

I want to send 50,000+ resquests to a host , in a ratelimited manner , So thought of using PipelineClient , cuz then i would be able to send more requests in Less Connections/Load to the server .

so while executing that , whenever i send more than 100-150 requests or so , randomly this EOF error comes . 2021/06/20 21:14:47 error in PipelineClient("157.245.125.234"): EOF EOF

Therefore , I wanted to know :--

  • What is this Error ,
  • and How can i avoid it ,
  • or Does it even matter because i am getting all of the responses back anyways , with hardly any time wasted . so just mute the error ? Edit : It does cause major delay sometimes .

Less than 100 -200:

root@snap:~/skrr# go run pipe.go 200
Sending  200 requests...

Time Taken : 995.608µs
Total Response Read := 200

root@snap:~/skrr# go run pipe.go 100
Sending  100 requests...

Time Taken : 602.138µs
Total Response Read := 100

More than 100-200:


root@snap:~/skrr# go run pipe.go 500
Sending  500 requests...
2021/06/20 21:28:21 error in PipelineClient("157.245.125.234"): EOF
EOF
Error Occured at Req No. : 251

Time Taken : 1.994905ms
Total Response Read := 500

root@snap:~/skrr# go run pipe.go 250
Sending  250 requests...
2021/06/20 21:29:22 error in PipelineClient("157.245.125.234"): EOF
EOF
Error Occured at Req No. : 125

Time Taken : 1.338302ms
Total Response Read := 250

Code : pipe.go

package main

import (
	"github.com/valyala/fasthttp" //nolint:goimports
	// "net/url"
	"reflect"
	"sync"
	"time"
	"os"
	"strconv"
	// "log"
	// "console"
	"fmt"
	"unsafe" //nolint:goimports
)

// var client *fasthttp.PipelineClient //nolint:gochecknoglobals

var batch int

func CreatePipelineClient(url string) *fasthttp.PipelineClient {
	return &fasthttp.PipelineClient { //nolint:gofmt
		Addr: url,
		MaxConns:            10000,
		MaxPendingRequests:  10000,
		MaxBatchDelay:       0,
		Dial:                nil,
		DialDualStack:       false,
		IsTLS:               false,
		TLSConfig:           nil,
		MaxIdleConnDuration: 50 * time.Second,
		ReadBufferSize:      4096,
		WriteBufferSize:     4096,
		ReadTimeout:         10 * time.Minute,
		WriteTimeout:        10 * time.Minute,
		Logger:              nil,
	}
}



func Request(uri string , c *int , cli *fasthttp.PipelineClient) {
	// fmt.Println(*c)
	// client := CreatePipelineClient("157.245.125.234")

	// if *c == batch {
		// cli = CreatePipelineClient("157.245.125.234")
		// fmt.Println("newclient at : ", batch)
		// batch = batch + 100
	// }
	
	

	destUrl := []byte(uri)

	req := fasthttp.AcquireRequest()
	res := fasthttp.AcquireResponse()

	req.SetRequestURIBytes(destUrl)


	if err := cli.Do(req, res); err != nil {
		println(err.Error())
		fmt.Printf("Error Occured at Req No. : %d \n", *c)
	}
	// time.Sleep(700 * time.Millisecond)

	fasthttp.ReleaseRequest(req)
	if res.StatusCode() == 200 {
		*c++
		// fmt.Println(*c)
		// fmt.Println(res.StatusCode() , BytesToString(destUrl)) // Print Response
		fasthttp.ReleaseResponse(res)
	}

}

func BytesToString(b []byte) string {
	bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
	sh := reflect.StringHeader{Data: bh.Data, Len: bh.Len}
	return *(*string)(unsafe.Pointer(&sh)) //nolint:wsl
}

// var syncMap sync.Map
// u, _ := url.Parse(ur)
// if _, ok := HTTPClients[u.Host]; !ok {
	// syncMap.Store(u.Host, CreatePipelineClient(u.Host))
	// HTTPClients[u.Host] = CreatePipelineClient(u.Host)
// }
// client := HTTPClients[u.Host]



// func pushnewclient(ur string) 

func main() {

	reqs := os.Args[1]
	// print
	re, _ := strconv.Atoi(reqs)
	// print(re)
	// os.Exit(0)
	wgfuzz := new(sync.WaitGroup)



	c := 0
	// count := &c

	start := time.Now()
	// limit := 100
	// lim := &limit
	fmt.Println("Sending " ,re , "requests...")
	client := CreatePipelineClient("157.245.125.234")
	
	for i := 0; i < re; i++ {

		wgfuzz.Add(1)
		// 

		go func() {	
			// Progress.TotalConn++
			// fmt.Println("connection No. : " , concount)
			Request("http://157.245.125.234" , &c , client)
			wgfuzz.Done()
		}()
	}

	elapsed := time.Since(start)

	wgfuzz.Wait()
	fmt.Printf("\nTime Taken : %s \nTotal Response Read := %d \n", elapsed , c )

}

Or maybe I made mistake in how pipelined request are suppossed to be sent . Help plz :/

erikdubbelboer
erikdubbelboer

Does the server you are sending the requests to support HTTP pipelining? Pipelining doesn't reduce the connections or load on the server, it will just send many requests at once. Normal servers always expect 1 request, then send the reply and then expect the next request. With pipelining you send many requests at once and then expect the replies for all of them.

Isn't the server just rate limiting you for sending too many requests?

Jun
21
2 days ago
Activity icon
created tag
createdAt 2 days ago
Activity icon
published release v1.27.0

erikdubbelboer in valyala/fasthttp create published release v1.27.0

createdAt 2 days ago
Jun
18
5 days ago
push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Increase timeouts for Windows github actions

commit sha: 874c8cad76f8bb0a0ee59cf7f749016e7b4cfcf4

push time in 4 days ago
push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Increase TestServerTLSReadTimeout timeout

And TestFSCompressConcurrent timeout

commit sha: 924a63fdeb4775ac2c8923ab9f5e47f8e6fa1ab9

push time in 4 days ago
push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Fix race condition in TestPipelineClientIssue832

commit sha: 410bde60095d39d9e976277addabd400ed667901

push time in 4 days ago
Activity icon
issue

erikdubbelboer issue comment valyala/fasthttp

erikdubbelboer
erikdubbelboer

Partial pipelined request blocks outstanding responses

If N pipelined requests are sent in the same packet, with the N+1th request split across a packet boundary. All replies are blocked until the N+1th request is finally sent. You can reproduce this behavior with the following client (run against helloworldserver example):

// socket client for golang
// https://golangr.com
package main

import "net"
import "fmt"
import "bufio"
import "time"

func main() {

  // connect to server
  conn, _ := net.Dial("tcp", "127.0.0.1:8080")
  reader := bufio.NewReader(conn)

  go func () {
    for { 
      message, _ := reader.ReadString('\n')
      fmt.Printf("%s reply from server: %s", time.Now(), message)
    }
  } ()
  payload := "GET / HTTP/1.1\r\nhost: localhost\r\n\r\nGE"
  fmt.Fprintf(conn, payload)
  fmt.Printf("%s client sent: %s", time.Now(), payload)
  time.Sleep(10 * time.Second) 
  
  payload = "T / HTTP/1.1\r\nhost: localhost\r\n\r\n"
  fmt.Fprintf(conn, payload)
  fmt.Printf("%s client sent: %s", time.Now(), payload)
  time.Sleep(10 * time.Second) 

}

I would expect the first request to respond immediately, instead it is blocked until the second request is fully parsed. I confirmed via tcpdump that no response packet is sent for a full 10 seconds.

Output for the above is the following:

2021-06-07 13:42:41.375146 -0700 PDT m=+0.002485389 client sent: GET / HTTP/1.1
host: localhost

GE2021-06-07 13:42:51.375805 -0700 PDT m=+10.003223534 client sent: T / HTTP/1.1
host: localhost

2021-06-07 13:42:51.376106 -0700 PDT m=+10.003523940 reply from server: HTTP/1.1 200 OK
2021-06-07 13:42:51.376213 -0700 PDT m=+10.003631153 reply from server: Server: fasthttp
2021-06-07 13:42:51.376222 -0700 PDT m=+10.003639915 reply from server: Date: Mon, 07 Jun 2021 20:42:41 GMT
2021-06-07 13:42:51.376228 -0700 PDT m=+10.003645898 reply from server: Content-Type: text/plain; charset=utf8
2021-06-07 13:42:51.376233 -0700 PDT m=+10.003651249 reply from server: Content-Length: 457
2021-06-07 13:42:51.376248 -0700 PDT m=+10.003666252 reply from server: X-My-Header: my-header-value
2021-06-07 13:42:51.37626 -0700 PDT m=+10.003678224 reply from server: Set-Cookie: cookie-name=cookie-value
2021-06-07 13:42:51.376266 -0700 PDT m=+10.003684119 reply from server: 
2021-06-07 13:42:51.376272 -0700 PDT m=+10.003690002 reply from server: Hello, world!
2021-06-07 13:42:51.376278 -0700 PDT m=+10.003695596 reply from server: 
2021-06-07 13:42:51.376286 -0700 PDT m=+10.003703790 reply from server: Request method is "GET"
2021-06-07 13:42:51.376292 -0700 PDT m=+10.003709599 reply from server: RequestURI is "/"
2021-06-07 13:42:51.376298 -0700 PDT m=+10.003716135 reply from server: Requested path is "/"
2021-06-07 13:42:51.376304 -0700 PDT m=+10.003721616 reply from server: Host is "localhost"
2021-06-07 13:42:51.37631 -0700 PDT m=+10.003728093 reply from server: Query string is ""
2021-06-07 13:42:51.376316 -0700 PDT m=+10.003733513 reply from server: User-Agent is ""
2021-06-07 13:42:51.376339 -0700 PDT m=+10.003756996 reply from server: Connection has been established at 2021-06-07 13:42:41.374607 -0700 PDT m=+1743.737173557
2021-06-07 13:42:51.37635 -0700 PDT m=+10.003767764 reply from server: Request has been started at 2021-06-07 13:42:41.375181 -0700 PDT m=+1743.737747673
2021-06-07 13:42:51.376355 -0700 PDT m=+10.003773139 reply from server: Serial request number for the current connection is 1
2021-06-07 13:42:51.37636 -0700 PDT m=+10.003778111 reply from server: Your ip is "127.0.0.1"
2021-06-07 13:42:51.376365 -0700 PDT m=+10.003783169 reply from server: 
2021-06-07 13:42:51.37637 -0700 PDT m=+10.003788145 reply from server: Raw request is:
2021-06-07 13:42:51.376375 -0700 PDT m=+10.003793212 reply from server: ---CUT---
2021-06-07 13:42:51.37638 -0700 PDT m=+10.003798132 reply from server: GET / HTTP/1.1
2021-06-07 13:42:51.376386 -0700 PDT m=+10.003803539 reply from server: Host: localhost
2021-06-07 13:42:51.376391 -0700 PDT m=+10.003808755 reply from server: 
2021-06-07 13:42:51.376397 -0700 PDT m=+10.003815315 reply from server: 
2021-06-07 13:42:51.376403 -0700 PDT m=+10.003820696 reply from server: ---CUT---HTTP/1.1 200 OK
2021-06-07 13:42:51.376408 -0700 PDT m=+10.003825883 reply from server: Server: fasthttp
2021-06-07 13:42:51.376417 -0700 PDT m=+10.003834568 reply from server: Date: Mon, 07 Jun 2021 20:42:51 GMT
2021-06-07 13:42:51.376423 -0700 PDT m=+10.003840440 reply from server: Content-Type: text/plain; charset=utf8
2021-06-07 13:42:51.37643 -0700 PDT m=+10.003847483 reply from server: Content-Length: 457
2021-06-07 13:42:51.376436 -0700 PDT m=+10.003853503 reply from server: X-My-Header: my-header-value
2021-06-07 13:42:51.376442 -0700 PDT m=+10.003860314 reply from server: Set-Cookie: cookie-name=cookie-value
2021-06-07 13:42:51.376448 -0700 PDT m=+10.003866027 reply from server: 
2021-06-07 13:42:51.376456 -0700 PDT m=+10.003873585 reply from server: Hello, world!
2021-06-07 13:42:51.376461 -0700 PDT m=+10.003879020 reply from server: 
2021-06-07 13:42:51.376467 -0700 PDT m=+10.003884499 reply from server: Request method is "GET"
2021-06-07 13:42:51.376522 -0700 PDT m=+10.003940311 reply from server: RequestURI is "/"
2021-06-07 13:42:51.376544 -0700 PDT m=+10.003962110 reply from server: Requested path is "/"
2021-06-07 13:42:51.376553 -0700 PDT m=+10.003971169 reply from server: Host is "localhost"
2021-06-07 13:42:51.37656 -0700 PDT m=+10.003977898 reply from server: Query string is ""
2021-06-07 13:42:51.376566 -0700 PDT m=+10.003983786 reply from server: User-Agent is ""
2021-06-07 13:42:51.376573 -0700 PDT m=+10.003990454 reply from server: Connection has been established at 2021-06-07 13:42:41.374607 -0700 PDT m=+1743.737173557
2021-06-07 13:42:51.376578 -0700 PDT m=+10.003996189 reply from server: Request has been started at 2021-06-07 13:42:51.375906 -0700 PDT m=+1753.738551447
2021-06-07 13:42:51.376584 -0700 PDT m=+10.004001803 reply from server: Serial request number for the current connection is 2
2021-06-07 13:42:51.37659 -0700 PDT m=+10.004007581 reply from server: Your ip is "127.0.0.1"
2021-06-07 13:42:51.376598 -0700 PDT m=+10.004016335 reply from server: 
2021-06-07 13:42:51.376605 -0700 PDT m=+10.004022976 reply from server: Raw request is:
2021-06-07 13:42:51.376611 -0700 PDT m=+10.004029062 reply from server: ---CUT---
2021-06-07 13:42:51.376622 -0700 PDT m=+10.004039669 reply from server: GET / HTTP/1.1
2021-06-07 13:42:51.376633 -0700 PDT m=+10.004050679 reply from server: Host: localhost
2021-06-07 13:42:51.376639 -0700 PDT m=+10.004056976 reply from server: 
2021-06-07 13:42:51.376648 -0700 PDT m=+10.004065554 reply from server: 
erikdubbelboer
erikdubbelboer

@wcs1only this is what I had in mind: https://github.com/valyala/fasthttp/pull/1050 Would you be willing to test this branch on your end to see if it completely fixes your problem?

pull request

erikdubbelboer pull request valyala/fasthttp

erikdubbelboer
erikdubbelboer

Flush buffered responses if we have to wait for the next request

Don't wait for the next request as this can take some time, instead flush the outstanding responses already.

Fixes #1043

Activity icon
created branch

erikdubbelboer in valyala/fasthttp create branch pipeline-flush

createdAt 4 days ago
Activity icon
issue

erikdubbelboer issue comment valyala/fasthttp

erikdubbelboer
erikdubbelboer

fix: set content-length properly when StreanRequestBody was enabled

Fix #1048

Please take a look. @valyala @kirillDanshin Thanks.

push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

fix: set content-length properly when StreanRequestBody was enabled (#1049)

  • fix: set content-length properly when StreanRequestBody was enabled

  • fix: add test cases for validating content length of streaming request

commit sha: 4ed933a2e770d457edba006bee30586c93d055df

push time in 4 days ago
pull request

erikdubbelboer pull request valyala/fasthttp

erikdubbelboer
erikdubbelboer

fix: set content-length properly when StreanRequestBody was enabled

Fix #1048

Please take a look. @valyala @kirillDanshin Thanks.

Activity icon
issue

erikdubbelboer issue valyala/fasthttp

erikdubbelboer
erikdubbelboer

Bug: incorrect content-length while enable StreamRequestBody.

The content length of a request is wrong when below conditions are fulfilled:

  • Enable StreamRequestBody
  • MaxRequestBodySize > real RequestBodySize
  • DisablePreParseMultipartForm

How to reproduce

Step1. Start a fasthttp server using this code

package main

import (
	"fmt"

	"github.com/valyala/fasthttp"
)

func main() {
	server := &fasthttp.Server{
		Handler: func(ctx *fasthttp.RequestCtx) {
			fmt.Println("req.content-legnth:", ctx.Request.Header.ContentLength())
		},
		DisablePreParseMultipartForm: true,
		MaxRequestBodySize:           10 * 1024 * 1024,
		StreamRequestBody:            true,
	}

	server.ListenAndServe(":8081")
}

Step2. Use curl to send request and ensure the request body size is less than 10M which is server's MaxRequestBodySize.

# dd if=/dev/zero of=5m.img bs=5m count=1
curl -F 'file=@5m.img' http://127.0.0.1:8081/

Then we see the server's output:

req.content-legnth: 8192

The content-length is always 8192 no matter how many requests are sent.

Possible cause

func (req *Request) ContinueReadBodyStream(r *bufio.Reader, maxBodySize int, preParseMultipartForm ...bool) error {
    // ignore unrelated code.

    bodyBuf := req.bodyBuffer()
    bodyBuf.Reset()
    bodyBuf.B, err = readBodyWithStreaming(r, contentLength, maxBodySize, bodyBuf.B)
    if err != nil {
        if err == ErrBodyTooLarge {
            req.Header.SetContentLength(contentLength)
            req.body = bodyBuf
            req.bodyStream = acquireRequestStream(bodyBuf, r, contentLength)
            return nil
        }
        if err == errChunkedStream {
            req.body = bodyBuf
            req.bodyStream = acquireRequestStream(bodyBuf, r, -1)
            return nil
        }
        req.Reset()
        return err
    }

    req.body = bodyBuf
    req.bodyStream = acquireRequestStream(bodyBuf, r, contentLength)
    req.Header.SetContentLength(len(bodyBuf.B)) // should be req.Header.SetContentLength(contentLength)
    return nil
}
Activity icon
issue

erikdubbelboer issue comment valyala/fasthttp

erikdubbelboer
erikdubbelboer

I can't shutdown my http server

fasthttp version: v1.23.0

Below is my test code, when I send a http request, server.GetOpenConnectionsCount value is not 1 , why ?

I cannot close my service because its number of connections is not zero

package main

import (
	"fmt"
	"github.com/valyala/fasthttp"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func RequestHandler(ctx *fasthttp.RequestCtx) {
	_, _ = fmt.Fprintf(ctx, "hello %s", "world")
	return
}

var stopCh = make(chan os.Signal)

func main() {
	server := &fasthttp.Server{
		Handler: RequestHandler,
	}
	signal.Notify(stopCh, syscall.SIGTERM, syscall.SIGINT)
	go func() {
		for {
			time.Sleep(1 * time.Second)
			fmt.Println(server.GetOpenConnectionsCount())
		}
	}()

	go func() {
		_ = server.ListenAndServe("0.0.0.0:8082")
	}()

	select {
	case <-stopCh:
		_ = server.Shutdown()
	}
}
erikdubbelboer
erikdubbelboer

You are correct, I have added IdleTimeout to the documentation for Shutdown. It will wait on both.

push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Add IdleTimeout to Shutdown documentation

commit sha: cec99537fed284176a808b49b1a16ac4802b572b

push time in 4 days ago
Activity icon
issue

erikdubbelboer issue comment valyala/fasthttp

erikdubbelboer
erikdubbelboer

TCPDialer :: DNSCacheDuration option

TCPDialer{} extended to support DNSCacheDuration

image

push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

TCPDialer :: DNSCacheDuration option (#1046)

  • DefaultDNSCacheDuration :: Changed to var

  • Revert "DefaultDNSCacheDuration :: Changed to var"

This reverts commit d836eace9f709b5993d73830aa12d5a2f548ce0d.

  • TCPDialer :: DNSCacheDuration option

  • TCPDialer :: DNSCacheDuration option - comment added

  • TCPDialer :: DNSCacheDuration option - comment fixed

commit sha: c12a06108b3a1d6b0a47d0635e66ebe0efc6ac9f

push time in 4 days ago
pull request

erikdubbelboer pull request valyala/fasthttp

erikdubbelboer
erikdubbelboer

TCPDialer :: DNSCacheDuration option

TCPDialer{} extended to support DNSCacheDuration

image

Activity icon
delete

erikdubbelboer in valyala/fasthttp delete branch github-test

deleted time in 4 days ago
push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Run go test on github actions (#1047)

  • Run go test on github actions

travis-ci.org has stopped. See also: https://github.com/curl/curl/issues/7150

Downside: github actions don't support ppc64le

  • Run less

  • delete .travis.yml

  • Remove travis + minor lint fixes

commit sha: 87fc95849cd9ec7ffb3555bd7ec6317865902241

push time in 4 days ago
pull request

erikdubbelboer pull request valyala/fasthttp

erikdubbelboer
erikdubbelboer

Run go test on github actions

travis-ci.org has stopped. See also: https://github.com/curl/curl/issues/7150

Downside: github actions don't support ppc64le

push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Remove travis + minor lint fixes

commit sha: 8a85a444d2742b9143fa7dd4972ab5533cbdb6d8

push time in 4 days ago
pull request

erikdubbelboer merge to valyala/fasthttp

erikdubbelboer
erikdubbelboer

fix: set content-length properly when StreanRequestBody was enabled

Fix #1048

Please take a look. @valyala @kirillDanshin Thanks.

erikdubbelboer
erikdubbelboer

Can you please add a test case that fails before the fix and passes after.

Jun
16
1 week ago
push

erikdubbelboer push valyala/fasthttp

erikdubbelboer
erikdubbelboer

Defined Transport for the client (#1045)

  • Defined Transport for the client

  • Added Transport test

commit sha: be13b504abc5fe227a924c0d8896b289c13b0379

push time in 6 days ago
Jun
15
1 week ago
pull request

erikdubbelboer pull request valyala/fasthttp

erikdubbelboer
erikdubbelboer

Run go test on github actions

travis-ci.org has stopped. See also: https://github.com/curl/curl/issues/7150

Downside: github actions don't support ppc64le

Activity icon
created branch

erikdubbelboer in valyala/fasthttp create branch github-test

createdAt 1 week ago