SourceHut Blocks Go Module Cache Proxy 'Go module mirror'

SourceHut , a code hosting service, announces that it will block from February 24, 2023 because the proxy `` Go Module Mirror '' that acquires and caches code from the Git server is using too much network bandwidth. Did.

SourceHut will blacklist the Go module mirror
https://SourceHut.org/blog/2023-01-09-gomodulemirror/

SourceHut to shun Google's Go Module Mirror over greed • The Register
https://www.theregister.com/2023/01/10/googles_go_SourceHut/

Go , a programming language developed by Google, retrieves modules from Git repositories, but each of these requests is routed through a caching proxy 'Go Module Mirror' located at 'proxy.golang.org'. Although the Go module mirror acquires caches to speed up Go module downloads, this Go Module Mirror periodically acquires Go packages from the origin source repository and checks for updates, resulting in a huge amount of traffic. There are reports of problems that can occur.

Go Module Mirror accounts for about 70% of outgoing network traffic from SourceHut's hosting server, git.sr.ht, and a single module can generate as much as 4GiB of traffic from Google in a single day. it is.

According to SourceHut, blocking Go Module Mirror was a last resort and he wanted to avoid it if possible, and reported the problem of excessive traffic to the Go development team on February 24, 2021. SourceHut and the Go development team discussed how to meet the requirements without generating excessive traffic, but in the end there was no effective solution, and Go Module Mirror continued to generate excessive traffic.

Drew Duveaux, CEO of SourceHut, said, 'We asked the public for a solution on May 25, 2022, but we couldn't find an effective solution, so we decided to block Go Module Mirror this time. The cost of incurring this traffic is no longer acceptable to us and the Go team has made no attempt to fix this issue.We don't want to inconvenience Go users, but this feature The burden and costs are too high to continue to justify supporting .

Other Go module maintainers also report increased traffic with Go Module Mirror. In an issue on Go's GitHub page, developer Ben Ladder said, ``Go Module Mirror downloaded 4GB of data from my server and requested one Go module more than 500 times. I'm the only one in the world who uses the module. I'd like to ask for caching or at least rate limiting,' he said while presenting the server log.

proxy.golang.org: Unusual traffic to git hosting service from Go Issue #44577 golang/go GitHub
https://github.com/golang/go/issues/44577#issuecomment-851079949

Russ Cox, head of Google's Go programming language technology, said on social news site HackerNews that the Go team is working to address the issue. According to Cox, Go 1.19 provides a way to download modules using the -reuse flag, which avoids unchanging data, allowing refresh operations to take up less bandwidth. thing. It seems that the proxy.golang.org service does not yet support this fix, but it seems that reflection work will be done in 2023.

Mr. Cox also mentioned that Duveau argued that ``Google must fix this problem,'' and that SourceHut should not receive special treatment, ``SourceHut , claims that this is a big problem for them, but also that they don't want the special treatment of disabling background updates for Go modules only in SourceHut. The suggestion of disabling background updates until we can deploy is still valid for SourceHut and everyone else struggling with the current load.'

・February 1, 2023 postscript
CEO Duveau updated his blog on January 31, 2023, revealing that SourceHut is canceling plans to disable Go Module Mirror traffic.

According to the communication Duveau received from Mr. Cox, adding the -reuse flag to Go will significantly reduce the traffic generated. Also, the Go development team acknowledged that the automatic update operation was their responsibility, and said that the automatic update traffic by Go Module Mirror was disabled for SourceHut. At the same time, the Go development team states that they can enable or disable the update behavior at their own discretion, so small hosts will not be affected.