rsc

rsc

Go Hacker. Mistake maker. (he/him)

Member Since 12 years ago

Google, Cambridge, MA

Experience Points
6.8k
follower
Lessons Completed
0
follow
Lessons Completed
3
stars
Best Reply Awards
128
repos

808 contributions in the last year

Pinned
⚡ The Go programming language
⚡ Fast, indexed regexp search over large file trees
⚡ Plan 9 from User Space
⚡ Packages and commands for using Plan 9 from Go
⚡ Two-factor authentication on the command line
Activity
Jan
19
5 days ago
Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: crypto/sha256: add native SHA256 instruction implementation for AMD64

The sha_ni sha256 instructions have been shown to provide an ~4x increase in hash rate on newer amd64 systems versus the avx2 implementation. Transliterating the Linux implementation shows an up to 3.79x increase in hash rate under benchmarks.

https://gist.github.com/monocasa/befeed9c8c5827417c0921e231703f2c

Q&A:

Q: Linux is GPL, is this OK to bring into golang?

A: It appears so. The Linux implementation file is dual BSD-3 clause/gpl. A look through the history of that file shows that all relevant changes have come from Intel employees, Intel has signed the CLA, and in fact the current AVX2 implementation submitted by Russ Cox is called out as arriving from the same source.

Q: Was the Assembly Policy followed?

A: I believe so, even to the point of leaving a measurable but ~1% perf increase on the table in exchange for simplifying constant table access and sharing that with the existing avx2 implementation.

rsc
rsc

Re licensing again, the other code in that file was contributed directly by Intel using Go's CLA in https://go-review.googlesource.com/22608, so we were not just "copying" the Linux version and therefore do not have to use that license. The right path might be to try to get Intel to contribute this one too.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: crypto/sha256: add native SHA256 instruction implementation for AMD64

The sha_ni sha256 instructions have been shown to provide an ~4x increase in hash rate on newer amd64 systems versus the avx2 implementation. Transliterating the Linux implementation shows an up to 3.79x increase in hash rate under benchmarks.

https://gist.github.com/monocasa/befeed9c8c5827417c0921e231703f2c

Q&A:

Q: Linux is GPL, is this OK to bring into golang?

A: It appears so. The Linux implementation file is dual BSD-3 clause/gpl. A look through the history of that file shows that all relevant changes have come from Intel employees, Intel has signed the CLA, and in fact the current AVX2 implementation submitted by Russ Cox is called out as arriving from the same source.

Q: Was the Assembly Policy followed?

A: I believe so, even to the point of leaving a measurable but ~1% perf increase on the table in exchange for simplifying constant table access and sharing that with the existing avx2 implementation.

rsc
rsc

Re licensing, the code needs to not introduce new notice requirements. What is the exact license on the code you are proposing to add? Also, if it's just an instruction we can use, it seems like we could write our own assembly and not worry about copying someone else's.

/cc @FiloSottile

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: reflect: add `Value.CallInto`

reflect.Call allocates a []reflect.Value slice of length nout for the number of return arguments from the function. This is fine for a one-off call of a function, but when repeatedly invoking a function, reflect.Call allocations can quickly add up.

As an example, I have a helper package that sorts arbitrary types, and if a type has a func (t T) Less(other T) bool (i.e., Name == "Less", NumIn() == 1, NumOut() == 1, In(0) == T, Out(0) == bool), the package calls that function for sorting. This is done within sort.Slice, meaning Less is called as many times as sort.Slice needs to compare. For an ordered slice of 1000 elements, each sort.Slice allocates 25735 times.

The only four allocations in this benchmark are (tip @2cf85b1fb8)

  105.50MB 68.28% 68.28%   105.50MB 68.28%  reflect.Value.call /home/twmb/go/go.tip/src/reflect/value.go:565
      41MB 26.53% 94.81%       41MB 26.53%  reflect.methodReceiver /home/twmb/go/go.tip/src/reflect/value.go:862
    4.50MB  2.91% 97.72%     4.50MB  2.91%  reflect.Value.call /home/twmb/go/go.tip/src/reflect/value.go:609
    1.50MB  0.97% 98.70%     1.50MB  0.97%  runtime.allocm /home/twmb/go/go.tip/src/runtime/proc.go:1866

70% of the allocations are from allocating the output argument slice. If possible, I could provide the input slice and reuse it for all Calls, which would effectively eliminate this alloc.

I looked into line 862 and into the runtime, I'm not sure how to get rid of that allocation: the code takes the address of an unsafe pointer, and that allocates; if possible it would be quite nice to get rid of this alloc as well but this would likely require deeper wiring into the runtime (the point here is to have a method pointer). Line 609 has a TODO that could eliminate the allocation. runtime/proc.go:1866 looks to be unrelated.

reflect.Value.call can be:

func (v Value) call(op string, in, out []Value) []Value {

If out is non-nil, then it is used rather than allocating a new slice. If the input out is too small, the code will panic. Essentially, line 565 turns into:

ret = out
if ret == nil {
    ret = make([]Value, nout)
}
if len(ret) < nout { // optional check
    panic("short output slice")
}
rsc
rsc

It still sounds like we don't have a final API to decide about, one that will guarantee zero allocations and is simple enough to present to users. Does anyone want to try to do that?

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: x/tools: move cmd/internal/edit to a public package

cmd/internal/edit is incredibly useful when modifying code, particular as compared to manually constructing an AST for go/printer to print. That's not surprising; it is the reason that Russ wrote it. I have a half dozen copies of cmd/internal/edit floating around, so that I can use it tools that I write.

Given its general utility when writing tools, I propose that we move cmd/internal/edit to x/tools, and then vendor it back into core.

rsc
rsc

My plan is to publish a copy under rsc.io.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: runtime: enable WER dumps for Windows under certain circumstances

CL 307372 was committed, which contained a nice change to produce crashdumps under certain circumstances. However, several questions were proposed about when those crashdumps would be automatically uploaded to Microsoft in the context of using Insider Builds. In the absence of good answers to that, the otherwise okay CL was reverted (by me) with CL 362454. This proposal is essentially about reverting the revert, and having an extended discussion about under which circumstances these should be enabled and whether we need a new flag for it, as suggested by @bufflig.

Before this proposal moves to an "under consideration" step, we should get a clearer idea about what we're proposing. I'll defer to @zhizheng052 and @bhiggins on this, as they wrote the original CL.

CC @zhizheng052 @bhiggins @bufflig @ianlancetaylor @alexbrainman @aarzilli @aclements

rsc
rsc

We are still waiting to find out exactly what is being proposed here. If we can't figure that out, it seems like this would be a likely decline.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: time: export zone and zoneTrans, and add a function to generate a Location

What version of Go are you using (go version)?

$ go version
go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/hugo/.cache/go-build"
GOENV="/home/hugo/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/hugo/.cache/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/hugo/.cache/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1388033309=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Trying to create an instance of time.Location based on data read from an iCalendar file.

What did you expect to see?

Some way to construct a time.Location instance.

What did you see instead?

No public function to create such an instance; time.Location instances can only be created with time.LoadLocationFromTZData.

More background

We're trying to parse iCalendar files, which contain VTIMEZONE entries, which represent a timezone (time.Location in go).

However, there's no public function to create time.Location instances, and the only way to create them is using using IANA Time Zone database-formatted data, which is a rather complex binary format.

So currently, the only way to create time.Location entries for the timezones read, is to convert them into the above mentioned binary format, and then feed that into time.LoadLocationFromTZData.

This has two big problems: (1) it's very complex to convert VTIMEZONE data into that binary format, and also, needless complexity (2) it's very inefficient, since every entry read has to be converted into an intermediate representation, to then be parsed again and then converted into the final one.

Making the time.zone and time.zoneTrans public (e.g.: renaming them into time.Zone and time.ZoneTrans, plus adding a time.LocationFromZoneInfo(name string, zones []Zone, zoneTrans []ZoneTrans) Location function would very much help address this issue.

I'd be happy to try and tackle the implementation myself, if the approach seems sound, but I figure it's best to ask for feedback on that first; it's possible that someone has cleaner approaches to solving this issue, or that this breaks some existing convention.

As an alternative approach, a function that takes a string with an RFC-5545 compliant VTIMEZONE and returns a time.Location instance might be a cleaner API, but I'm not sure if adding iCalendar-specific code into this package would be deemed acceptable.

rsc
rsc

Seems like we are still waiting to find the answer to:

If you have a vtimezone2tzdata, then it seems like that could be a 3rd-party package used in concert with time.LoadLocationFromTZData, and then we don't need to add code to package time at all.

Do I have that right?

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: x/time/rate: expose the number of remaining tokens

rate.Limiter holds the remaining tokens in a bucket here https://github.com/golang/time/blob/master/rate/rate.go#L59

The feature request is to expose this value in a read only way, perhaps as a public function Tokens.

The use case is that when using a limiter it would be nice to be able to see how many tokens remain without attempting to use one. For my specific case, I want to emit a metric from my kubernetes controller indicating how many tokens are left in the client side rate limiter for a kubernetes clientset (which uses a rate.Limiter under the hood). I think this should be useful in other situations as well though.

Open to any other workarounds as well!

I'd be happy to contribute this, the Tokens func is probably just

// Tokens returns the remaining tokens in the bucket
func (lim *Limiter) Tokens() float64 {
	lim.mu.Lock()
	defer lim.mu.Unlock()
	lim.advance(time.Now())
	return lim.tokens
}

I'll create a real PR (and get my contribution environment set up) if this idea sounds reasonable. Thanks!

rsc
rsc

Does anyone object to the API in https://github.com/golang/go/issues/50035#issuecomment-1013416864?

@ryanmcnamara does that API solve your use case? Thanks.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: spec: safe multidimensional array conversion

Background

As of today, if one wishes to obtain backing slice for a multidimensional arrays one must use unsafe. Example (also see go playground)

mat := [2][2]float64{
    {1, 2},
    {3, 4},
}
backing := (*[4]float64)(unsafe.Pointer(&mat[0][0]))
backingAlternative := (*[4]float64)(unsafe.Pointer(&mat))

Unsafe gets the job done but provides two disadvantages:

  1. Marks widespread use of these conversions as unsafe. Today lots of code depends on the row-major ordering of elements in multidimensional arrays. It would be nice to standarize this behaviour.
  2. Unsafe breaks some transpiler tools such as gopherjs

Proposal

For better numerical programming facilities, such as those in gonum. It'd be handy to convert between matrix (multidimensional array) form and backing slice form without the use of unsafe. Example of how this could be implemented in the language:

type backer *[4]float64
mat := [2][2]float64{ {1, 2}, {3, 4} }
backing := backer(&mat)

The code above eliminates the need to use unsafe.

Other benefits: When working with single dimension arrays that represent matrices, one could also convert to multidimensional form for better bounds checking

type matForm *[2][2]float64
unidim := [4]float64{1, 2, 3, 4 }
mat := matForm(&unidim)
mat[1][0] = 0
mat[2][1] = 3 // Compiler error!

Final notes

I wonder if it would be possible to convert in a single line

mat := [2][2]float64{ {1, 2}, {3, 4} }
backing := (*[4]float64)(&mat) // is this too much?
rsc
rsc

It doesn't seem like the situation has changed since https://github.com/golang/go/issues/50118#issuecomment-1011343076.

@kortschak, we understand that you can't do, say, a *[3][3]int to *[9]int conversion today without unsafe, but you can do it, by using unsafe. Using unsafe is OK.

The big question is how often this arises. We decided the slice to array pointer conversion happened enough that it was worth adding the special case. The problem here is that the need for compile-time-constant array sizes means that it seems like this doesn't happen enough. And it's still possible with unsafe when it does happen.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: net: make LookupCNAME consistent between Unix and Windows, document

LookupCNAME is pretty weird right now.

Despite the name, it entirely ignores CNAME records on Unix. It launches A and AAAA record lookups to recursive resolvers and returns the first response name found in the A and AAAA, skipping over any CNAME. (and not even asking for a CNAME)

But it documents that it does that...

https://pkg.go.dev/net#LookupCNAME

A canonical name is the final name after following zero or more CNAME records. LookupCNAME does not return an error if host does not contain DNS "CNAME" records, as long as host resolves to address records.

OTOH, on Windows, it does what you would expect from the name itself: it looks up CNAME records:

func (*Resolver) lookupCNAME(ctx context.Context, name string) (string, error) {
        // TODO(bradfitz): finish ctx plumbing. Nothing currently depends on this.
        acquireThread()
        defer releaseThread()
        var r *syscall.DNSRecord
        e := syscall.DnsQuery(name, syscall.DNS_TYPE_CNAME, 0, nil, &r, nil)

Here's a demo of a program behaving differently:

func main() {
	txt, err := net.LookupTXT("cname-to-txt.go4.org")
	log.Printf("LookupTXT = %q, %v", txt, err)

	cname, err := net.LookupCNAME("cname-to-txt.go4.org")
	log.Printf("cname = %q, %v", cname, err)
}

On Linux/Mac:

2021/12/10 21:19:45 LookupTXT = ["foo=bar"], <nil>
2021/12/10 21:19:45 cname = "", lookup cname-to-txt.go4.org: no such host

On Windows:

2021/12/10 21:11:45 LookupTXT = ["foo=bar"], <nil>
2021/12/10 21:11:45 cname = "test-txt-record.go4.org.", <nil>

I like the Windows behavior better, FWIW. That's what I was looking for, but apparently it doesn't exist.

Can we either:

  1. add LookupCNAMERecord that actually looks up a CNAME record
  2. redefine LookupCNAME to be like Windows, perhaps adding a LookupCanonicalName with the current weird Unix behavior of LookupCNAME?

But at minimum: document whatever the rules are and make Unix and Windows match? At least in Resolver.PreferGo mode?

rsc
rsc

Sounds like we are still waiting for someone to check what can be done on the Mac.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: archive/tar: add FileInfoHeaderNoNames

Abstract

archive/tar function FileInfoHeader does uid -> uname and gid -> name lookups, which are not always necessary and can sometimes be problematic. A new function, FileInfoHeaderNoNames, is proposed to address these issues.

Background

Change https://go-review.googlesource.com/59531 (which made its way to Go 1.10) implemented user/group name lookups in tar/archive's FileInfoHeader. It fills in tar file info header fields Uname and Gname, looking up user and group names (from Uid and Gid) via os/user.LookupId and LookupGroupId functions.

Doing that is not always desirable, and is sometimes problematic:

  1. In a chrooted environment, /etc/passwd and /etc/group may be absent, or their contents may be entirely different from that of the host.

  2. Failed name lookups are not currently cached, which may result in a considerable performance regression, caused by re-parsing of /etc/passwd and /etc/group for every file entry added to the tar.

  3. In case of static linking against glibc, the latter wants to dlopen some libraries that might either be unavailable (which results in a panic/crash) or (in case of untrusted chroot) a malicious library can be substituted by a bad actor.

  4. There may be a need to create a tarball without any user/group names (only with numeric uids/gids), akin to GNU tar's --numeric-owner option.

  5. There may be a need to use custom uid -> name and gid -> name lookup functions.

Now, problem 2 can be mitigated by using (indirectly, via os/user Lookup{,Group}Id) a good C library that does caching, or by caching failed lookups as well. Problem 3 can be solved by using osusergo build tag, but it's compile unit wide, meaning it will also affect other os/user uses, not just archive/tar. Yet it seems impossible to solve both 2 and 3 at the same time.

As far as I know, there are no easy solutions for problems 1 and 5.

In particular, this affects Docker, which performs image unpacking by re-executing the main binary (dockerd) in the container context (essentially a chroot). As a workaround, Docker maintains a fork of archive/tar with commit 0564e304a6ea partially reverted (see https://github.com/moby/moby/issues/42402).

Proposal

Add a function similar to FileInfoHeader, which does not perform any id -> name lookups, leaving it to a user. The proposed name is FileInfoHeaderNoNames (can also be *NoLookup, *Num, etc).

Rationale

Adding a new function seems to be the most simple and elegant approach, with very little code to add, and yet solving all the issues raised above.

Alternatives are:

  • (for users) to maintain the fork of archive/tar
  • (for archive/tar) to implement a build tag which disable lookups (e.g. archivetarnolookups or archivetarnumeric)
  • (for archive/tar) to add a way to provide own name lookup function
  • (for archive/tar) to add another way to disable lookups (so a user can do tar.NameLookup = false or tar.NameLookup(false)
  • to unconditionally remove id -> name lookups from archive/tar (might bring compatibility issues)

Compatibility

Since this is a new API, and the existing functionality of FileInfoHeader is left intact, there are no compatibility issues.

Implementation

See https://go-review.googlesource.com/c/go/+/371054 for the example code.

rsc
rsc

It sounds like people are mostly happy with https://github.com/golang/go/issues/50102#issuecomment-1011336173, and I've come around to FileInfoNames as a name for that limited interface. We should embed FileInfo, though:

type FileInfoNames interface {
    fs.FileInfo
    Uname() string
    Gname() string
}

Does anyone object to this?

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: time: export Location.lookup() as Location.Lookup()

Export Location.lookup() as Location.Lookup() to help finding DST transitions.

Sometimes one want to find the when a zone transition starts or ends, which is already implemented in the non exported func (l *Location) lookup(sec int64) (name string, offset int, start, end int64, isDST bool)

Example: I need to correct '2021-03-28 02:30:00' in 'Europe/Amsterdam', which is in the non-existing hour during DST transition between CET and CEST, to the first coming valid time. I.e. there is a requirement to have a function that takes a date and time in a DST Location and if it is incorrect, correct it to the first following correct date and time.

Setting the time and then reading it back may give a new time that is not at the DST transition, like: https://go.dev/play/p/sJdTyNZJPT6

And I don't expect time.Date() to change it current behaviour, but if there was an efficient way to find the DST transition one could easily just check the DST transition and correct it one self.

rsc
rsc

It sounds like the current ZoneBounds proposal is:

// ZoneBounds returns the bounds of the time zone in effect at time t.
// The zone begins at start and the next zone begins at end.
// If the zone begins at the beginning of time, start will be returned as a zero Time.
// If the zone goes on forever, end will be returned as a zero Time.
// The Location of the returned times will be the same as t.
func (t Time) ZoneBounds() (start, end Time)

(The difference from iant's comment above is that end is a zero time when there isn't one.)

Do I have that right? Does anyone object to that?

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: time: export Location.lookup() as Location.Lookup()

Export Location.lookup() as Location.Lookup() to help finding DST transitions.

Sometimes one want to find the when a zone transition starts or ends, which is already implemented in the non exported func (l *Location) lookup(sec int64) (name string, offset int, start, end int64, isDST bool)

Example: I need to correct '2021-03-28 02:30:00' in 'Europe/Amsterdam', which is in the non-existing hour during DST transition between CET and CEST, to the first coming valid time. I.e. there is a requirement to have a function that takes a date and time in a DST Location and if it is incorrect, correct it to the first following correct date and time.

Setting the time and then reading it back may give a new time that is not at the DST transition, like: https://go.dev/play/p/sJdTyNZJPT6

And I don't expect time.Date() to change it current behaviour, but if there was an efficient way to find the DST transition one could easily just check the DST transition and correct it one self.

rsc
rsc

The Extend string seems like too much. Those are a cryptic, not terribly well-defined format, and we don't want to expose that as our API for all time.

But adding ZoneBounds seems OK.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: syscall/js: Reinstate js.Wrapper interface

js.Wrapper was removed when removing support for js.Wrapper in js.ValueOf() in 6c0daa7331920. In #39740 @finnbear suggested writing this proposal after I asked about the decision in https://github.com/golang/go/issues/39740#issuecomment-998777142 and https://github.com/golang/go/issues/39740#issuecomment-999160824.

I understand the reasoning for removing the case Wrapper: from js.ValueOf(), which makes good sense.

What I regret is the removal of the interface itself, it served a purpose besides the case in js.ValueOf(). It allowed packages and functions to signal very easily and precisely that they accept a "type backed by an underlying javascript object.".

js.Wrapper was perfect for this purpose, it provided glue for packages in the Go wasm community. We could write packages like this, and inter-package calling was well-defined:

package a

import (
	"syscall/js"
)

func DoStuffInJavascriptLand(obj js.Wrapper) {
	js.Global().Call("some_js_function", obj.JSValue())
}

Users of package a could satisfy this interface either by embedding js.Value or by satisfying the interface in another way. It was very convenient and ensured that all packages agreed allowing simple inter-package calling.

I propose the following:

  • Reinstate the js.Wrapper interface.
  • Reinstate the JSValue() method on js.Value.

This will solve this case of inter-package calling in the Go wasm community, and we can avoid packages defining their own (incompatible) interfaces for this purpose.

rsc
rsc

Referring back to my last comment, we have:

(1) js.ValueOf cannot contain a test for a JSValue method [performance] (2) adding a JSValue method to js.Value would really imply that js.ValueOf should use it.

The implication then seems to be that we shouldn't add a JSValue method to js.Value.

Especially since @abrander says this is not causing any problems right now, let's leave it out for Go 1.18. We can always add something later.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: os/exec: add fields for managing termination signals and pipes

Background

#23019 (accepted but not yet implemented; CC @ianlancetaylor @bradfitz) proposed to change exec.Cmd.Wait to stop the goroutines that are copying I/O to and from a completed exec.Cmd; see that proposal for further background on the problem it aims to address. However, as noted in https://github.com/golang/go/issues/23019#issuecomment-396372555 and https://github.com/golang/go/issues/23019#issuecomment-558416418, any feasible implementation of the proposal requires the use of an arbitrary timeout, and the proposal does not include a mechanism to adjust that timeout. (Given our history with the Go project's builders, I am extremely skeptical that any particular hard-coded timeout can strike an appropriate balance between robustness and latency.)

#31774, #22757, and #21135 proposed to allow users of exec.CommandContext to customize the signal sent to the command when the context is canceled. They were all declined due to lack of concrete demand for the feature (https://github.com/golang/go/issues/21135#issuecomment-332006025, https://github.com/golang/go/issues/22757#issuecomment-345034265, https://github.com/golang/go/issues/31774#issuecomment-490246274). We have since accrued a number of copies of functions that work around the feature's absence. In the Go project alone, we have:

I'm attempting to add yet another variation (in CL 373005) in order to help diagnose #50014. However, for this variation (prompted by discussions with @aclements and @prattmic) I have tried to make this variation a minimally-invasive change on top of the exec.Cmd API.

I believe I have achieved that goal: the API requires the addition of only 2–3 new fields and no new methods or top-level functions. You can view (and try out) a prototype as github.com/bcmills/more/os/moreexec, which provides a drop-in replacement for a subset of the exec.Cmd API.

Proposal

I propose the addition of the following fields to the exec.Cmd struct, along with their corresponding implementation:

	// Context is the context that controls the lifetime of the command
	// (typically the one passed to CommandContext).
	Context context.Context

	// If Interrupt is non-nil, Context must also be non-nil and Interrupt will be
	// sent to the child process when Context is done.
	//
	// If the command exits with a success code after the Interrupt signal has
	// been sent, Wait and similar methods will return Context.Err()
	// instead of nil.
	//
	// If the Interrupt signal is not supported on the current platform
	// (for example, if it is os.Interrupt on Windows), Start may fail
	// (and return a non-nil error).
	Interrupt os.Signal

	// If WaitDelay is non-zero, the command's I/O pipes will be closed after
	// WaitDelay has elapsed after either the command's process has exited or
	// (if Context is non-nil) Context is done, whichever occurs first.
	// If the command's process is still running after WaitDelay has elapsed,
	// it will be terminated with os.Kill before the pipes are closed.
	//
	// If the command exits with a success code after pipes are closed due to
	// WaitDelay and no Interrupt signal has been sent, Wait and similar methods
	// will return ErrWaitDelay instead of nil.
	//
	// If WaitDelay is zero (the default), I/O pipes will be read until EOF,
	// which might not occur until orphaned subprocesses of the command have
	// also closed their descriptors for the pipes.
	WaitDelay time.Duration

The new Context field is exported only in order to simplify the documentation for the Interrupt and WaitDelay fields. (It was requested and rejected in #46699, but the objection there was my own — due to concerns about the interactions with the API in this proposal. It could be excised from this proposal without damaging anything but documentation clarity.)

The new Interrupt field sets the signal to be sent when the Context is done. exec.CommandContext explicitly sets it to os.Kill in order to maintain the existing behavior of exec.CommandContext, but I expect many users on Unix platforms will want to set it to os.Interrupt or syscall.SIGQUIT instead.

The new WaitDelay field sets the interval to wait for input and output after process termination or an interrupt signal. That interval turns out to be important for many testing applications (such as the Go Playground implementation and the cmd/go test suite). It also generalizes nicely to the use-cases in #23019: setting WaitDelay without Context provides bounded I/O wait times without sending a preceding signal.

Compatibility

I believe that this proposal is entirely backward-compatible (in contrast with #23019). The zero-values for the new fields provide exactly the same behavior as a Cmd returned by exec.Command or exec.CommandContext today.

Caveats

This proposal does not address graceful shutdown on Windows (https://github.com/golang/go/issues/22757#issuecomment-773581159; CC @mvdan). However, it may be possible to extend it to do so by providing special-case Windows behavior when the Interrupt field is set to os.Interrupt, or by adding an InterruptFunc func(*Cmd) callback that would also be invoked when Context is done.

The proposed API also does not provide a mechanism to send an Interrupt signal followed by os.Kill after a delay but still wait for subprocesses to close all I/O pipes. I believe the use-cases for that scenario are sufficiently niche to be provided only by third-party libraries: sending SIGKILL to the parent process makes it likely that subprocesses will not know to shut down, so in the vast majority of cases users should either not send SIGKILL at all (WaitDelay == 0), forcibly terminate the pipes to try to kill the subprocesses with SIGPIPE (WaitDelay > 0), or do something platform-specific to try to forcibly shut down an entire process group (outside the scope of this proposal).

Alternatives considered

In https://github.com/golang/go/issues/31774#issuecomment-488320719, @bradfitz suggested a field Kill func(*os.Process), which would presumably be added instead of the Interrupt field in this proposal. However, I believe that such a field would be simultaneously too complex and not powerful enough:

  • The Kill field would be too complex for most Unix applications, which overwhelmingly only need to send one of SIGTERM, SIGINT, SIGQUIT, or SIGKILL — why pass a whole callback when you really just want to say which signal you need?

  • A *os.Process callback would still not be powerful enough for Windows applications. If I understand the discussion in #6720 correctly (CC @alexbrainman), CTRL_BREAK_EVENT is sent to an entire process group, not a single *os.Process, so Windows users would also need a mechanism for creating (or determining) such a group, or some completely separate out-of-band way to request that the process terminate (such as by sending it a particular input or IPC message).

Given the above, the Interrupt field seems more ergonomic: it gives the right behavior for Unix users, and if Windows users want to do something more complex they can set Interrupt to nil and start a separate goroutine in between the calls to (*Cmd).Start and (*Cmd).Wait to implement whatever custom logic they want.

rsc
rsc

It seems like the docs need to distinguish what we call cmd.Stdin from cmd.Stdout/cmd.Stderr. cmd.Stdin is already closed when we finish writing whatever was standard input, independent of exiting/waiting/cancellation.

I'm also confused by the WaitDelay description mixing cancellation and ordinary exits.

Maybe it would help to see the diff that would implement this? I looked at CL 373005 but it's a whole package, not a diff.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: cmd/go: Need an easy/obvious way to copy testdata from module cache to a test-running directory

To test, benchmark, or fuzz-test downloaded software modules, the current recipe (at least, the one that I learned) is

mkdir foo
cd foo
go mod init foo
go get -d -t -v [email protected]
go test -c -o foo somepackage
./foo -test.bench=BenchmarkSomething

The problem here is that test and benchmarking often depend on files in testdata. It is often possible to run the test in the module cache, but this depends on the test/benchmark not writing to the testdata directory, and also depends on knowing the directory in the module cache (see workaround below). Fuzzing that finds a failure does write to the testdata directory, so this just doesn't work for new fuzzing.

One workaround is

cp -rp `go list -f {{.Dir}} somepackage`/testdata .

but this is not something that people find or figure out easily.

I propose adding an option to go get or a subcommand to go mod to copy testdata, if it exists in the downloaded package, into the current directory. For example, go get -d -t -T -v [email protected] or go mod testdata.

rsc
rsc

When you run go test foo it runs in the module directory. Why not do that? Why copy the data out of the module directory? In general it's not clear exactly what to copy. For example compress/gzip etc use the compress/testdata directory. So in general it's not 1:1.

You can do what go test does using cd $(go list -f '{{.Dir}}' pkg).

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: cmd/go: Add a `-C`flag to `go` - change dir

I'm not sure if this rises to a proposal or not.

Many tools (e.g. make, git) have a universal -C <dir> flag which lets you do things in other directories without pushd/popd silliness. Given how go uses CWD for important things like modules, it would be nice to have something like this in go. For example go -C path/to/dir list gives me the package-path. Otherwise it's things like pushd path/to/dir >/dev/null; go list; popd >/dev/null.

It's really just a convenience thing. Has it already been discussed? I couldn't find it.

rsc
rsc

If -C is really "do a chdir before anything else", then that's easy. But what about

go build -C bar -o x.exe 

or worse

go build -o x.exe -C bar

Does that x.exe get written to bar/x.exe? What behavior will people expect?

It sounds like the suggestion here is that this command writes to bar/x.exe. Will that be too confusing for people?

/cc @matloob @bcmills

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: sort: add a boolean-valued search function

I have never used sort.Search to build a sorted list (slice). It's too expensive and there are better ways. However, I use it often to do what its name suggests: search.

The API is designed for the growth case, which is fine and general, but the general (growth) case is rare, perhaps very rare, and handling the return value is clumsy†.

I suggest adding another function, say sort.Contains, that just says yes or no to the question of presence of the element. Here is sort.Search's use for the pure search case, right from the documentation:

i := sort.Search(len(data), func(i int) bool { return data[i] >= x })
        if i < len(data) && data[i] == x {
        	// x is present at data[i]
        } else {
        	// x is not present in data,
        	// but i is the index where it would be inserted.
        }

Besides the messy handling, that extra comparison has always bothered me.

Here is what that would look like with the proposed function:

 if sort.Contains(len(data), func(i int) bool { return data[i] >= x }) {
      	// x is present at data[i]
}

This seems so much more intuitive and semantically lightweight. If you needed the index - but why would you if you weren't going to insert there? - you could go back to the existing Search function.

It should be very simple to implement by having both Search and Contains call a helper that reports presence, and returning the appropriate result, or by just writing Contains itself - like Search, it's very short.

I already filled in the proposal questionnaire here: https://github.com/golang/go/issues/45624

† And poorly documented, but that is fixable.

rsc
rsc

When we do resolve this discussion, perhaps we should revisit https://github.com/golang/go/issues/47619#issuecomment-915428658.

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

proposal: sort: add a boolean-valued search function

I have never used sort.Search to build a sorted list (slice). It's too expensive and there are better ways. However, I use it often to do what its name suggests: search.

The API is designed for the growth case, which is fine and general, but the general (growth) case is rare, perhaps very rare, and handling the return value is clumsy†.

I suggest adding another function, say sort.Contains, that just says yes or no to the question of presence of the element. Here is sort.Search's use for the pure search case, right from the documentation:

i := sort.Search(len(data), func(i int) bool { return data[i] >= x })
        if i < len(data) && data[i] == x {
        	// x is present at data[i]
        } else {
        	// x is not present in data,
        	// but i is the index where it would be inserted.
        }

Besides the messy handling, that extra comparison has always bothered me.

Here is what that would look like with the proposed function:

 if sort.Contains(len(data), func(i int) bool { return data[i] >= x }) {
      	// x is present at data[i]
}

This seems so much more intuitive and semantically lightweight. If you needed the index - but why would you if you weren't going to insert there? - you could go back to the existing Search function.

It should be very simple to implement by having both Search and Contains call a helper that reports presence, and returning the appropriate result, or by just writing Contains itself - like Search, it's very short.

I already filled in the proposal questionnaire here: https://github.com/golang/go/issues/45624

† And poorly documented, but that is fixable.

rsc
rsc

It sounds like we agree that sort.Search is hard to use, but there's not an obvious replacement. In particular, anything that can report an exact position has to take a different function (a 3-way result like strings.Compare, or multiple functions).

For sake of argument, we could do:

func Find(n int, cmp func (i int) int) int

where the cmp function returns <0, 0, or >0 to say whether the item being sought is less than, equal to, or greater than element at position i.

(In comparison, sort.Search's f reports whether the item being sought is less than the element at position i; that is, it's Less(item, elem[i]), whereas Find takes Cmp(item, elem[i]).)

Is that something we should consider adding?

Activity icon
issue

rsc issue comment golang/go

rsc
rsc

internal/poll: deadlock in Read on arm64 when an FD is closed [1.17 backport]

@bcmills requested issue #45211 to be considered for backport to the next 1.17 minor release.

@gopherbot, please backport to 1.17 and 1.16: this is a race condition that can result in deadlocks on ARM64 in any process that relies on Close unblocking a read. It is tricky to diagnose (and hard to reproduce reliably), and there is no apparent workaround for users on ARM64 machines.

rsc
rsc

I'd feel better about a backport if we had more data showing that the crashes really have stopped.

push

rsc push ossf/osv-schema

rsc
rsc

Add optional top-level severity field (#21)

The new severity field allows declaring quantitative severity metrics. For now the only one is CVSS_V3.

commit sha: fd9c3420a5950269d2f7dfc6ef45592d8df7d5bf

push time in 5 days ago
pull request

rsc pull request ossf/osv-schema

rsc
rsc

Add optional top-level severity field

This adds a top-level severity array field field to capture vulnerability severity rankings such as CVSS. Each severity is an object with a type enum and a score.

Initially supported types are CVSS_V3 and PLAIN_TEXT, described in the schema documentation.

pull request

rsc merge to ossf/osv-schema

rsc
rsc

Add optional top-level severity field

This adds a top-level severity array field field to capture vulnerability severity rankings such as CVSS. Each severity is an object with a type enum and a score.

Initially supported types are CVSS_V3 and PLAIN_TEXT, described in the schema documentation.

Activity icon
issue

rsc issue comment ossf/osv-schema

rsc
rsc

Add optional top-level severity field

This adds a top-level severity array field field to capture vulnerability severity rankings such as CVSS. Each severity is an object with a type enum and a score.

Initially supported types are CVSS_V3 and PLAIN_TEXT, described in the schema documentation.

rsc
rsc

The new one with just CVSS_V3 looks like a good start. Thanks for putting "quantitative" in the bottom table entry.

Not for this PR, but one thing I would support in the future if it were valuable to GH or others is some standard enumeration of vulnerability categories, like "denial of service", "crash", "remote code execution", and so on. Listing the worst-case possibilities in that form might help users adapt their response appropriately.

Jan
18
6 days ago
Activity icon
issue

rsc issue comment ossf/osv-schema

rsc
rsc

Add optional top-level severity field

This adds a top-level severity array field field to capture vulnerability severity rankings such as CVSS. Each severity is an object with a type enum and a score.

Initially supported types are CVSS_V3 and PLAIN_TEXT, described in the schema documentation.

rsc
rsc

For the CRITICAL/HIGH/MEDIUM/LOW it would still be good to have a sense of what these mean, to keep the entries precise. More generally each type we define should be backed by a definition, either in this spec or linked to from this spec. Is there some agreed-upon industry-wide meaning for these? (Somehow I expect not.) Are there databases that define their own meanings for these? If so, maybe we should have a type named for the database that defines the scale, or perhaps add the field to database_specific?

Activity icon
created branch

rsc in rsc/plan9vmware create branch main

createdAt 6 days ago
Activity icon
created repository

rsc in rsc/plan9vmware create repository

createdAt 6 days ago
Jan
17
1 week ago
Activity icon
issue

rsc issue golang/go

rsc
rsc

cmd/compile: error for unexpected return values does not say 'return'

p1% cat /tmp/x.go
package p

func f0_2()            { return 1, 2 }
func f0_1()            { return 1 }
func f1_0() int        { return }
func f1_2() int        { return 1, 2 }
func f2_0() (int, int) { return }
func f2_1() (int, int) { return 1 }
p1% go tool compile /tmp/x.go
/tmp/x.go:3:33: no result values expected   <<<
/tmp/x.go:4:33: no result values expected   <<<
/tmp/x.go:5:26: not enough return values
	have ()
	want (int)
/tmp/x.go:6:36: too many return values
	have (number, number)
	want (int)
/tmp/x.go:7:26: not enough return values
	have ()
	want (int, int)
/tmp/x.go:8:33: not enough return values
	have (number)
	want (int, int)
p1% 

The marked messages that say 'result values' should be rewritten to say 'return values', both for consistency with the others and for clarity.

The simplest fix is probably to stop treating f0_1 and f0_2 as a special case and let the general 'too many return values' printer handle them, just like f1_0 and f2_0` are not special cases and are handled by the general 'not enough return values' printer.

I know there's still plenty of generics work left in the type checker. I can take care of this one myself this week.

Jan
16
1 week ago
Activity icon
delete

rsc in rsc/ivy delete branch pfor

deleted time in 1 week ago
push

rsc push ossf/osv-schema

rsc
rsc

Add credits field (#22)

Add a top-level credits field, with a list of contacts specified by URLs.

commit sha: 84a1491f448408d21aecb50172c76b587c70b439

push time in 1 week ago
pull request

rsc pull request ossf/osv-schema

rsc
rsc

Add credits field

This proposes a top-level credits field. We learned through internal research projects that credits are an important form of currency in the vulnerability community, and often a strong motivator for identifying or fixing vulnerabilities. It provides the benefit of name recognition as well as acts as a resume of sorts for vulnerability researchers. In order to proved proper credit, it's important that contributors are able to be represented with both a name as well as unique identifiers or ways to link back to other work they have done.

Because of this, we chose to implement credits as a list of objects rather than strings for a few reasons:

  1. Contributors may not want their personal email addresses published, so we don't want to rely on simple lists of "First Last <email...>" values.
  2. Different source databases may use different identifiers to represent contributing users
  3. Contributors may wish to include multiple contact methods, or none
  4. Processing systems may wish to process contact information dynamically

With those limitations in mind, we can define credits similar to how references are defined.