Skip to content
This repository has been archived by the owner on Mar 26, 2022. It is now read-only.

Totals calculation #35

Open
ArcticCascade opened this issue Mar 6, 2017 · 18 comments
Open

Totals calculation #35

ArcticCascade opened this issue Mar 6, 2017 · 18 comments

Comments

@ArcticCascade
Copy link

3.0.4 has a great update to calculating the totals.
I'm just curious as to why I would be seeing such a large difference between what cahcer and my server logs say.

2017-03-06 12:53:20.673 Since server start: 7.09 GB returned to clients, 1.46 GB stored from Internet, 0 bytes from peers; 0 bytes imported.

Cacher has retrieved the following stats for 2017-03-06:
4.8 GB of bandwith served to client devices.
1.4 GB of bandwith requested from Apple
0.0 bytes of bandwith requested from other Caching Servers

My server only runs between 6am and 11pm each day, so there's no crossover between dates.
@erikng
Copy link
Owner

erikng commented Mar 6, 2017 via email

@erikng
Copy link
Owner

erikng commented Mar 6, 2017 via email

@ArcticCascade
Copy link
Author

From the server log:

2017-03-06 15:31:20.732 Since server start: 8.79 GB returned to clients, 1.95 GB stored from Internet, 0 bytes from peers; 0 bytes imported.

From cacher 3.0.3:

8.79 Gigabytes of bandwith served to client devices.

From cacher 3.0.4:

5.8 GB of bandwith served to client devices.

"Only runs between" means the server (the mac-mini) shuts down at 11pm every night and boots up again at 6am - so "Since server start" will always (excepting any reboots during the day) total from 6am that day.

@erikng
Copy link
Owner

erikng commented Mar 6, 2017

Ping @Hunty1

@calum-github
Copy link
Contributor

the new calculation method in my opinion is far more accurate.
It looks at every GET request from client devices and adds how much data was served to the client device.

I have no idea how apple was calculating the information from the "Since server start: "

Perhaps that includes dropped data, imported data, put data etc etc

@calum-github
Copy link
Contributor

You would have to look at the logs for that period of time and see how apple comes up with that figure.

@erikng
Copy link
Owner

erikng commented Mar 6, 2017 via email

@calum-github
Copy link
Contributor

Well it depends what figure you want. If you want the figures from the actual GET requests - the new commit is accurate.

@ArcticCascade you will need to attach the logs for the period you claim the difference in data.

Also doesn't 3.0.3 still do duplication on numbers because it looks at service logs as well as debug logs?

Cacher/cacher.py

Lines 848 to 853 in c7c4edd

# Now combine all .log files in the destination into a temp file that's
# removed when python exits
rawLog = tempfile.TemporaryFile()
for anyLog in glob.glob(os.path.join(tmpLogs, '*.log')):
with open(anyLog, 'rb') as f:
shutil.copyfileobj(f, rawLog)

My runs are showing the same data in metrics db 'bytes.fromorigin.toclients' as per cacher 3.0.4
So i'd probably trust that more than Since server start up

I've almost finished the work on adding an option to use either logs or the metrics db for measurement of bandwidth.

@ArcticCascade
Copy link
Author

Thanks guys. I'll upload the logs when I get back to work tomorrow.

@erikng
Copy link
Owner

erikng commented Mar 6, 2017

@Hunty1 while it is indeed duplicated, the first and last event will still be the correct data (according to since Server start). Regardless, let's wait.

@ArcticCascade please also post the db so we can compare.

@ArcticCascade
Copy link
Author

Caching Logs.zip
Sorry for the delay. These logs should cover everything from March 6th.
I've included metrics.sqlite - I assume thats the db you mean.

@ArcticCascade ArcticCascade reopened this Mar 7, 2017
@ArcticCascade
Copy link
Author

I don't think there's anything wrong with your algorithm. I've taken a small sample from this morning's logs (Since server start: 369.4 MB) and added them up in Excel and come out with a total of 337.9 MB (matching cacher's results). Apple appears to be counting something else that's not logged in the usual way.

@calum-github
Copy link
Contributor

Apple might be counting the HEAD requests

@calum-github
Copy link
Contributor

calum-github commented Mar 7, 2017 •
edited
Loading

eg:

2017-03-08 10:18:24.731 #DSjgqKuqhJ2b Received HEAD request from 153.107.193.13:18198 [com.apple.appstored/1.0 iOS/10.2.1 model/iPhone9,1 hwp/t8010 build/14D27 (6; dt:137)] for /apple-assets-us-std-000001/Purple111/v4/0c/af/81/0caf8180-e64c-01e5-d678-4bbf414f17ca/mzps.2282574410900846017.ipa

2017-03-08 10:18:24.765 #DSjgqKuqhJ2b Served 0 bytes of 22.0 MB; 0 bytes from cache, 0 bytes stored from Internet, 0 bytes from peers

@ArcticCascade
Copy link
Author

That can't be it. In my sample, there is a total of 611.3 MB requested.
That doesn't match either total served value.

@erikng
Copy link
Owner

erikng commented Mar 7, 2017 via email

@calum-github
Copy link
Contributor

That is quite possible. I am not using decimal for my calculations.

https://github.com/hunty1/Cacher/blob/master/cacher.py#L172-L181

@erikng Perhaps we should change this to decimal?

@ArcticCascade
Copy link
Author

I don't think so.
With the numbers I'm seeing, the difference between base 10 and base 2 would be negligible.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@erikng @calum-github @ArcticCascade