crypto/sha256: optimize sha256 implementation

[WangLeonard]

What version of Go are you using (go version)?

$ go version
`go version go1.12.4 linux/amd64`

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

GOARCH="amd64"  
lsb_release -a

No LSB modules are available.

Distributor ID:	Ubuntu

Description:	Ubuntu 16.04.6 LTS

Release:	16.04

Codename:	xenial

and my cpu is Intel(R) Xeon(R) E5-2690 v3 @ 2.60GHz(support avx2)

openssl version is OpenSSL 1.0.2g  1 Mar 2016

What did you do?

I test the performance in the following three cases:

1. Call the openssl library with C
2. Call the crypto library with Go
3. Call the openssl library with Go's Cgo

For different length strings, do 10,000,000 sha256 calculations.
The result is as follows, the unit of data is second.

length of string	openssl	golang	cgo
1	2.93	4.47	4.43
16	2.96	4.43	4.38
128	6.93	9.21	8.53
511	18.77	23.95	21.89
512	18.37	23.14	20.63
1023	34.08	42.51	37.38
1024	33.61	41.74	36.76
4096	125.56	152.72	133.35
8192	248.05	301.27	262.73

What did you expect to see?

Test results in different cases are similar.

What did you see instead?

It can be seen that when the length of the string is small, openssl is more than 30% faster than golang.
When string length is long, using cgo to call openssl is also 10% faster than golang.

Here is my demo (does not contain loops and time statistics), please indicate if there is an error, thank you.

Call the openssl library with C (need to install openssl-dev)

#include <stdio.h>
#include <openssl/sha.h>
#include <string.h>

void sha256String(char *string, unsigned char *digest)
{
    SHA256_CTX sha256;
    SHA256_Init(&sha256);
    SHA256_Update(&sha256, string, strlen(string));
    SHA256_Final(digest, &sha256);
}
int main() {
    unsigned char digest[SHA256_DIGEST_LENGTH];
    sha256String("a",digest);
    return 0;
}
// gcc -c -g -o sha256.o sha256.c
// gcc sha256.o -lssl -lcrypto -g -O0 -o sha256

golang and cgo

package main

//#cgo LDFLAGS:-lssl -lcrypto
//#include <stdio.h>
//#include <openssl/sha.h>
//#include <string.h>
//
//void sha256String(char *string, unsigned char *digest,int strlen)
//{
//SHA256_CTX sha256;
//SHA256_Init(&sha256);
//SHA256_Update(&sha256, string, strlen);
//SHA256_Final(digest, &sha256);
//}
import "C"
import (
	"crypto/sha256"
	"fmt"
	"unsafe"
)

func sha256StringCgo(b []byte)[]uint8{
	output:=make([]uint8,32,32)
	inputa:=(*C.char)(unsafe.Pointer(&b[0]))
	inputb:=(*C.uchar)(unsafe.Pointer(&output[0]))
	inputc:=C.int(len(b))
	C.sha256String(inputa,inputb,inputc)
	return output
}

func sha256StringGo(b []byte)[]uint8{
	h:=sha256.New()
	h.Write(b)
	ans:=h.Sum(nil)
	return ans
}

func main(){
	fmt.Println(sha256StringCgo([]byte("a")))
	fmt.Println(sha256StringGo([]byte("a")))
}

So can we optimize the sha256 of golang? Because this is a hot function in my application, I am concerned about the time efficiency of sha256.

Here are some preliminary analysis, I hope it can provide some help.
The longest running function in the test, in golang is $GOROOT/src/crypto/sha256/sha256block_amd64.s block avx2 case,
in openssl is sha256_block_data_order_avx2.
You can download openssl 1.0.2g source file, make , then can found it in openssl-1.0.2g/crypto/sha/sha256-x86_64.s.
sha256-x86_64.s.zip

Since I am not very familiar with assembly, I don't know how to optimize.
Look forward to your reply, thank you.

[katiehockman]

/cc @FiloSottile

[FiloSottile]

The Go standard library places higher value on reducing complexity, so it doesn't aim to be the fastest at implementing primitives. If this makes it unsuitable for your use case, please do share some details, as we take that into account when deciding how much complexity to take. See also https://golang.org/design/cryptography-principles.

There are more aggressively optimized third-party implementations of SHA-256 you might want to consider instead, like https://github.com/minio/sha256-simd.

[WangLeonard]

Ok, I will find a computer that support for avx512 to verify the performance of sha256-simd, Thank you very much.