siddontang

siddontang

Member Since 10 years ago

PingCAP, China

Experience Points
2.7k
follower
Lessons Completed
0
follow
Lessons Completed
422
stars
Best Reply Awards
140
repos

11 contributions in the last year

Pinned
⚡ a MySQL proxy powered by Go
⚡ my golang lib
⚡ a fast distributed message queue implemented with go
⚡ libtnet is a tiny high performance c++ network lib, like tornado
⚡ Elasticsearch note
Activity
Nov
20
1 week ago
close pull request

siddontang wants to merge tikv/tikv

siddontang
siddontang

raftstore: increase batch for raftlog-gc-worker (release-5.1-20211115)

What problem does this PR solve?

Issue Number: close https://github.com/tikv/tikv/issues/11404

What is changed and how it works?

Batch more deleted keys together. As we all know ,the deleted key must be much small than the entry which has been put. So 256 is too small for every writebatch in raftlog-gc-worker. And the gc tasks of most of regions are small. They are often smaller than 100. So I batch the keys of multiple regions and write them together.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • PR to update pingcap/tidb-ansible:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM
  • Breaking backward compatibility

Release note

None
siddontang
siddontang

can we try delete_range to speed up this?

pull request

siddontang merge to tikv/tikv

siddontang
siddontang

raftstore: increase batch for raftlog-gc-worker (release-5.1-20211115)

What problem does this PR solve?

Issue Number: close https://github.com/tikv/tikv/issues/11404

What is changed and how it works?

Batch more deleted keys together. As we all know ,the deleted key must be much small than the entry which has been put. So 256 is too small for every writebatch in raftlog-gc-worker. And the gc tasks of most of regions are small. They are often smaller than 100. So I batch the keys of multiple regions and write them together.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • PR to update pingcap/tidb-ansible:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM
  • Breaking backward compatibility

Release note

None
Oct
20
1 month ago
Activity icon
issue

siddontang issue comment tikv/tikv

siddontang
siddontang

*: remove prost support

What problem does this PR solve?

Problem Summary: Since it's not used in production, remove it to reduce dependencies and codes.

To fully remove prost, we still need pprof to support protobuf-codec.

See also #10905.

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Release note

None
siddontang
siddontang

awesome, this can simplify our code a lot. I hope we can advance this.

/cc @zhangjinpeng1987

Sep
26
2 months ago
close pull request

siddontang wants to merge cloudwego/netpoll

siddontang
siddontang

doc: fix guide code demo

siddontang
siddontang

I'm very interested at debug see eventLoop.svr is nil after runtime.gc , how do you know this variable has not been persisted and be garbage collected.

close pull request

siddontang wants to merge cloudwego/netpoll

siddontang
siddontang

doc: fix guide code demo

siddontang
siddontang

I suggest adding the comments about why do we need a global var for eventLoop here.

I guess the reason likes https://blog.csdn.net/idwtwt/article/details/102985319

Activity icon
issue

siddontang issue comment cloudwego/netpoll

siddontang
siddontang

seem it can't work on Mac OSX

I try to write a simple test on Mac OSX below, but it can't work. I use telnet 127.0.0.1 8080, type hello world, but can't get the response.

package main

import (
	"context"
	"flag"
	"fmt"
	"time"

	"github.com/cloudwego/netpoll"
)

var (
	addr = flag.String("addr", "127.0.0.1:8080", "server address")
)

func onRequest(_ context.Context, conn netpoll.Connection) error {
	println("hello world")

	var reader, writer = conn.Reader(), conn.Writer()

	defer reader.Release()
	// client send "Hello World", size is 11
	buf, err := reader.Next(11)
	if err != nil {
		fmt.Printf("error %v\n", err)
		return err
	}

	alloc, _ := writer.Malloc(len(buf))
	copy(alloc, buf)
	err = writer.Flush()
	if err != nil {
		fmt.Printf("flush error %v\n", err)
		return err
	}

	return nil
}

func onPrepare(conn netpoll.Connection) context.Context {
	println("hello prepare")
	return context.TODO()
}

func main() {
	flag.Parse()
	l, err := netpoll.CreateListener("tcp", *addr)
	if err != nil {
		panic("create netpoll listener failed")
	}
	defer l.Close()

	println("hello event loop")

	loop, err1 := netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err1 != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
}
Activity icon
issue

siddontang issue comment cloudwego/netpoll

siddontang
siddontang

seem it can't work on Mac OSX

I try to write a simple test on Mac OSX below, but it can't work. I use telnet 127.0.0.1 8080, type hello world, but can't get the response.

package main

import (
	"context"
	"flag"
	"fmt"
	"time"

	"github.com/cloudwego/netpoll"
)

var (
	addr = flag.String("addr", "127.0.0.1:8080", "server address")
)

func onRequest(_ context.Context, conn netpoll.Connection) error {
	println("hello world")

	var reader, writer = conn.Reader(), conn.Writer()

	defer reader.Release()
	// client send "Hello World", size is 11
	buf, err := reader.Next(11)
	if err != nil {
		fmt.Printf("error %v\n", err)
		return err
	}

	alloc, _ := writer.Malloc(len(buf))
	copy(alloc, buf)
	err = writer.Flush()
	if err != nil {
		fmt.Printf("flush error %v\n", err)
		return err
	}

	return nil
}

func onPrepare(conn netpoll.Connection) context.Context {
	println("hello prepare")
	return context.TODO()
}

func main() {
	flag.Parse()
	l, err := netpoll.CreateListener("tcp", *addr)
	if err != nil {
		panic("create netpoll listener failed")
	}
	defer l.Close()

	println("hello event loop")

	loop, err1 := netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err1 != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
}
siddontang
siddontang

em, amazing, can work now...

Activity icon
issue

siddontang issue comment cloudwego/netpoll

siddontang
siddontang

seem it can't work on Mac OSX

I try to write a simple test on Mac OSX below, but it can't work. I use telnet 127.0.0.1 8080, type hello world, but can't get the response.

package main

import (
	"context"
	"flag"
	"fmt"
	"time"

	"github.com/cloudwego/netpoll"
)

var (
	addr = flag.String("addr", "127.0.0.1:8080", "server address")
)

func onRequest(_ context.Context, conn netpoll.Connection) error {
	println("hello world")

	var reader, writer = conn.Reader(), conn.Writer()

	defer reader.Release()
	// client send "Hello World", size is 11
	buf, err := reader.Next(11)
	if err != nil {
		fmt.Printf("error %v\n", err)
		return err
	}

	alloc, _ := writer.Malloc(len(buf))
	copy(alloc, buf)
	err = writer.Flush()
	if err != nil {
		fmt.Printf("flush error %v\n", err)
		return err
	}

	return nil
}

func onPrepare(conn netpoll.Connection) context.Context {
	println("hello prepare")
	return context.TODO()
}

func main() {
	flag.Parse()
	l, err := netpoll.CreateListener("tcp", *addr)
	if err != nil {
		panic("create netpoll listener failed")
	}
	defer l.Close()

	println("hello event loop")

	loop, err1 := netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err1 != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
}
siddontang
siddontang

yes, tried but still can't work, below

	var loop netpoll.EventLoop
	loop, err = netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
Activity icon
issue

siddontang issue comment cloudwego/netpoll

siddontang
siddontang

seem it can't work on Mac OSX

I try to write a simple test on Mac OSX below, but it can't work. I use telnet 127.0.0.1 8080, type hello world, but can't get the response.

package main

import (
	"context"
	"flag"
	"fmt"
	"time"

	"github.com/cloudwego/netpoll"
)

var (
	addr = flag.String("addr", "127.0.0.1:8080", "server address")
)

func onRequest(_ context.Context, conn netpoll.Connection) error {
	println("hello world")

	var reader, writer = conn.Reader(), conn.Writer()

	defer reader.Release()
	// client send "Hello World", size is 11
	buf, err := reader.Next(11)
	if err != nil {
		fmt.Printf("error %v\n", err)
		return err
	}

	alloc, _ := writer.Malloc(len(buf))
	copy(alloc, buf)
	err = writer.Flush()
	if err != nil {
		fmt.Printf("flush error %v\n", err)
		return err
	}

	return nil
}

func onPrepare(conn netpoll.Connection) context.Context {
	println("hello prepare")
	return context.TODO()
}

func main() {
	flag.Parse()
	l, err := netpoll.CreateListener("tcp", *addr)
	if err != nil {
		panic("create netpoll listener failed")
	}
	defer l.Close()

	println("hello event loop")

	loop, err1 := netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err1 != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
}
siddontang
siddontang

Em, I don't think this is the reason. I have tested this on Linux and it works well.

For the above test, I add println in onPrepare and onRequest but find no output.

Activity icon
issue

siddontang issue cloudwego/netpoll

siddontang
siddontang

seem it can't work on Mac OSX

I try to write a simple test on Mac OSX below, but it can't work. I use telnet 127.0.0.1 8080, type hello world, but can't get the response.

package main

import (
	"context"
	"flag"
	"fmt"
	"time"

	"github.com/cloudwego/netpoll"
)

var (
	addr = flag.String("addr", "127.0.0.1:8080", "server address")
)

func onRequest(_ context.Context, conn netpoll.Connection) error {
	println("hello world")

	var reader, writer = conn.Reader(), conn.Writer()

	defer reader.Release()
	// client send "Hello World", size is 11
	buf, err := reader.Next(11)
	if err != nil {
		fmt.Printf("error %v\n", err)
		return err
	}

	alloc, _ := writer.Malloc(len(buf))
	copy(alloc, buf)
	err = writer.Flush()
	if err != nil {
		fmt.Printf("flush error %v\n", err)
		return err
	}

	return nil
}

func onPrepare(conn netpoll.Connection) context.Context {
	println("hello prepare")
	return context.TODO()
}

func main() {
	flag.Parse()
	l, err := netpoll.CreateListener("tcp", *addr)
	if err != nil {
		panic("create netpoll listener failed")
	}
	defer l.Close()

	println("hello event loop")

	loop, err1 := netpoll.NewEventLoop(
		onRequest,
		netpoll.WithReadTimeout(time.Second),
		netpoll.WithOnPrepare(onPrepare),
	)
	if err1 != nil {
		panic("create event loop failed")
	}

	println("begin to serve")
	loop.Serve(l)
}
Sep
25
2 months ago
started
started time in 2 months ago
Sep
10
2 months ago
started
started time in 2 months ago