BTFS-2419 #782

NathanQiu666 · 2020-11-11T03:53:34Z

No description provided.

codecov · 2020-11-11T04:01:17Z

Codecov Report

Merging #782 (5ed3870) into master (8deeac3) will decrease coverage by 0.03%.
The diff coverage is 3.70%.

@@            Coverage Diff             @@
##           master     #782      +/-   ##
==========================================
- Coverage   11.62%   11.58%   -0.04%     
==========================================
  Files         149      150       +1     
  Lines       13111    13317     +206     
==========================================
+ Hits         1524     1543      +19     
- Misses      11276    11466     +190     
+ Partials      311      308       -3

Impacted Files	Coverage Δ
cmd/btfs/daemon.go	`0.00% <0.00%> (ø)`
core/commands/storage/challenge/challenge.go	`0.00% <0.00%> (ø)`
core/commands/storage/challenge/dc_challenge.go	`4.32% <4.69%> (ø)`
core/commands/wallet.go	`0.00% <0.00%> (-4.59%)`	⬇️
core/corehttp/gateway_indexPage.go	`77.77% <0.00%> (-4.58%)`	⬇️
...ore/commands/storage/upload/upload/upload_shard.go	`0.00% <0.00%> (ø)`
core/wallet/tron.go	`41.07% <0.00%> (+0.44%)`	⬆️
core/coreapi/unixfs.go	`2.60% <0.00%> (+1.54%)`	⬆️
core/corehttp/gateway_handler.go	`42.66% <0.00%> (+1.97%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac6214b...baae414. Read the comment docs.

JustinGaoF

没完全看懂，可能我有些理解不太对，当前我的问题是：

S3 只能通过url来获取，这个文件是否一定存在s3上实际上我们不保证的，而且我也不太确认让所有客户都装s3的驱动是否合适
我没看到调用入口，好像你是把接受challenge question这个命令复用了，实际上就现在设计，这个是得由host自己发起请求来获取文件。

JustinGaoF · 2021-01-15T01:15:32Z

core/commands/storage/challenge/challenge.go

@@ -41,49 +41,12 @@ still store a piece of file (usually a shard) as agreed in storage contract.`,
 	}, StorageChallengeResponseCmd.Arguments...), // append pass-through arguments
 	RunTimeout: 20 * time.Second,
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
-		cfg, err := cmdenv.GetConfig(env)
+		scr, err := ReqChallengeStorage(req, res, env, req.Arguments[0], req.Arguments[2], req.Arguments[3], req.Arguments[4], req.Arguments[5]) //(*StorageChallengeRes, error) {


Not fully sure for this part, my understanding is decentral challenge will start another command, it seems the command you are changing still need in the future. But I am not sure.

The function is involved in the spin package, quite like heart beat, challengers request challenge jobs for every 60 mins.

还是没搞懂你为啥要改写这个命令，感觉上这个命令未来还是得用的，我们的router node还是得做challenge的，如果这个decentral challenge没成功的话。

另外，这个地方你每60分钟启动一次可能有问题；理论上这个challenge应该是完成工作以后就继续要下一个，60分钟最多check一下如果出了问题去trigger一下。

As discussed, it is ok for now.

JustinGaoF · 2021-01-15T01:18:13Z

core/commands/storage/challenge/dc_challenge.go

+		return nil, fmt.Errorf("failed to reuqest challenge job for peer id: {%s}", n.Identity.Pretty())
+	}
+
+	rawData, err := repo.DefaultS3Connection.RetrieveData(challengeResp.PackageUrl)


Go-BTFS client cannot use s3 connection to retrieve the file, it should use the http connection to get the file.

Update the code to use http request to get all the questions.

JustinGaoF · 2021-01-15T01:26:59Z

core/commands/storage/challenge/dc_challenge.go

+	challengeJobResult := &guardpb.ChallengeJobResult{
+		NodePid:    peerId,
+		JobId:      challengeResp.JobId,
+		Result:     challengeResults,


不太确认，好像你这个代码没有block等待的逻辑，所以你生成这个object的时候，challengeResults 应该是没有内容的。

Optimized the code and control the number of go routine, and make sure all the challenge jobs have been done and then submit the challenge results.

JustinGaoF

To solve fileHash issue, I had update the file's schema, please review that PR and fix it accordingly.
Still not fully understand some code, and doubt it will cause the issue.

JustinGaoF · 2021-01-27T00:09:30Z

core/commands/storage/challenge/challenge.go

@@ -41,49 +41,12 @@ still store a piece of file (usually a shard) as agreed in storage contract.`,
 	}, StorageChallengeResponseCmd.Arguments...), // append pass-through arguments
 	RunTimeout: 20 * time.Second,
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
-		cfg, err := cmdenv.GetConfig(env)
+		scr, err := ReqChallengeStorage(req, res, env, req.Arguments[0], req.Arguments[2], req.Arguments[3], req.Arguments[4], req.Arguments[5]) //(*StorageChallengeRes, error) {


还是没搞懂你为啥要改写这个命令，感觉上这个命令未来还是得用的，我们的router node还是得做challenge的，如果这个decentral challenge没成功的话。

JustinGaoF · 2021-01-27T00:10:36Z

core/commands/storage/challenge/challenge.go

@@ -41,49 +41,12 @@ still store a piece of file (usually a shard) as agreed in storage contract.`,
 	}, StorageChallengeResponseCmd.Arguments...), // append pass-through arguments
 	RunTimeout: 20 * time.Second,
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
-		cfg, err := cmdenv.GetConfig(env)
+		scr, err := ReqChallengeStorage(req, res, env, req.Arguments[0], req.Arguments[2], req.Arguments[3], req.Arguments[4], req.Arguments[5]) //(*StorageChallengeRes, error) {


另外，这个地方你每60分钟启动一次可能有问题；理论上这个challenge应该是完成工作以后就继续要下一个，60分钟最多check一下如果出了问题去trigger一下。

JustinGaoF · 2021-01-27T00:38:41Z

core/commands/storage/challenge/dc_challenge.go

+	resultChan := make(chan *guardpb.ShardChallengeResult, len(questions))
+	for _, question := range questions {
+		requestChan <- struct{}{}
+		go doChallenge(req, res, env, question, requestChan, resultChan)


This part start many go routine , if the questions count is high, it will exceeds the memory.

Btw, wg is not good to be defined as static variable, why just define before the for loop?

I used the length of channel to control the number of go routine.
ok, wg can be local variable and pass to go method.

JustinGaoF · 2021-01-27T00:45:39Z

core/commands/storage/challenge/dc_challenge.go

+		ShardHash: question.ShardHash,
+		Nonce:     question.Nonce,
+	}
+	go func() {


honestly, not get understanding why start another go routine here, why just run the retry function directly and wait for its response, I noticed challengeHostBo already has timeout control.

Simplified the code here.

JustinGaoF · 2021-01-27T01:12:08Z

core/commands/storage/challenge/dc_challenge.go

+		return nil, fmt.Errorf("receive wrong status code %d response: %s", resp.StatusCode, string(rawData))
+	}
+
+	var questions []*Question


I did some code change today for challenge question to add fileHash, so that you can use proto.decentralQuestions to unmarshall the file. And to reduce the bandwidth, I use gzip to zip the file. The PR is here.

https://github.com/TRON-US/go-btfs-guard/pull/507/

ok, code changed to read gzip file.

thanks, code changed to read and unmarshal gzip file.

BTFS-2419

a50039e

dc-challenge-branch

bb79d2b

NathanQiu666 requested a review from JustinGaoF December 18, 2020 01:05

JustinGaoF reviewed Jan 15, 2021

View reviewed changes

“NathanQiu666” added 9 commits January 20, 2021 21:57

dc-challenge-branch

e8ed8b0

dc-challenge-branch

c59f8f8

dc-challenge-branch

5d8713c

dc-challenge-branch

457b39b

dc-challenge-branch

1a80d78

dc-challenge-branch

5672920

dc-challenge-branch

a88b608

dc-challenge-branch

bb719a2

dc-challenge-branch

0795548

JustinGaoF suggested changes Jan 27, 2021

View reviewed changes

“NathanQiu666” added 5 commits January 27, 2021 20:46

dc-challenge-branch

3ee7953

dc-challenge-branch

75d6752

dc-challenge-branch

5ed3870

dc-challenge-branch

4567849

dc-challenge-branch

baae414

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BTFS-2419 #782

BTFS-2419 #782

NathanQiu666 commented Nov 11, 2020

codecov bot commented Nov 11, 2020 •

edited

Loading

JustinGaoF left a comment

JustinGaoF Jan 15, 2021

NathanQiu666 Jan 20, 2021 •

edited

Loading

JustinGaoF Jan 27, 2021

JustinGaoF Jan 27, 2021

NathanQiu666 Jan 27, 2021

JustinGaoF Jan 15, 2021

NathanQiu666 Jan 20, 2021

JustinGaoF Jan 15, 2021

NathanQiu666 Jan 20, 2021

JustinGaoF left a comment

JustinGaoF Jan 27, 2021

JustinGaoF Jan 27, 2021

JustinGaoF Jan 27, 2021

NathanQiu666 Jan 27, 2021 •

edited

Loading

JustinGaoF Jan 27, 2021

NathanQiu666 Jan 27, 2021

JustinGaoF Jan 27, 2021

NathanQiu666 Jan 27, 2021

NathanQiu666 Jan 27, 2021

BTFS-2419 #782

Are you sure you want to change the base?

BTFS-2419 #782

Conversation

NathanQiu666 commented Nov 11, 2020

codecov bot commented Nov 11, 2020 • edited Loading

Codecov Report

JustinGaoF left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanQiu666 Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JustinGaoF left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanQiu666 Jan 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 11, 2020 •

edited

Loading

NathanQiu666 Jan 20, 2021 •

edited

Loading

NathanQiu666 Jan 27, 2021 •

edited

Loading