Too many reports received (startDate and/or endDate ignored?)

[olivluca]

No matter the value I put in startDate (I first removed the *7 to get just one day, then I tried 3600000, i.e. an hour) I seem to receive always all the reports.
(I mean, I changed

data = '{"search": [{"endDate": %d, "startDate": %d, "ids": %s}]}' % (unixEpoch *1000, (unixEpoch *1000) -(86400000 *7), ids.keys())

first to

data = '{"search": [{"endDate": %d, "startDate": %d, "ids": %s}]}' % (unixEpoch *1000, (unixEpoch *1000) -(86400000), ids.keys())

then to

data = '{"search": [{"endDate": %d, "startDate": %d, "ids": %s}]}' % (unixEpoch *1000, (unixEpoch *1000) -3600000, ids.keys())
)

I don't know if it's a problem with apple servers or the parameters aren't properly formatted, I checked and apparently it's the same format used in the mobile application of openhaystack.

[biemster]

Odd stuff. I never tinkered with this, but I noticed now receiving reports from 12 days ago with these default values.

[olivluca]

Well, no matter what I put in startDate and endDate (I tried using cocoa timestamps instead of unix timestamps, seconds instead of microseconds, nanoseconds instead of seconds, 0, etc.) I always get back all the available reports.

[biemster]

Leaving them out completely will result in a lot less reports (40 for me atm)..

[olivluca]

I think I tried that too, let me check....

[biemster]

it's actually max 20 per tag if you leave out the date fields I see now. In my last test there were 2 tags with reports.
So it seems that with the fields it returns max 2000 reports per hash, without the fields 20.

[olivluca]

Mmh, 2383 reports received with StartDate/EndDate, 1688 without.
I have 100 virtual tags.
Also, the filtering isn't working right: I requested reports with -H 1 and the first one is from 3 days ago.

[biemster]

Are there any tags in your batch of 100 that have more than 20 reports, when calling without the startDate/endDate?

[olivluca]

Scratch that, my virtual mac still thinks it's monday 😉

[olivluca]

Since I rotate the key every hour usually I get far less than 20 reports per tag.

[olivluca]

In any case the StartDate/EndDate values must be wrong, since even with those present I get many reports before the StartDate

[olivluca]

i.e., now that my mac is on the correct time, with -H 1 I (and StartDate/EndDate) I get 2380 reports, 0 used (my tag has been off since yesterday night).

[biemster]

Yeah it's about time the OpenHaystack people came back to work, the academic year started two weeks ago lazy bums. @dakhnod asked there as well a while ago seemoo-lab/openhaystack#137 but they don't seem to have answered to issues all summer.

[olivluca]

I tried:

• formating the StartDate as isodatetime
• as YYYYMMDDHHNNSS
• as a raw unix timestampo
• as the same multiplied by 1000000
• multiplied by 1000000000
• multiplied by 1000000000000

and I always get the same number of reports and they are more than if I leave the StartDate field out altogether.
I also left out the EndDate, guessing that if it isn't there it should give you until the last report received.

[olivluca]

If I'm looking at the right file file it seems that openhaystack is using a unix timestamp multiplied by 1000 (I don't know if the format modifier %lli adds some zeroes), I tried it too (the "1000" multiplier was missing from my last round of tests) but the result is the same.
In any case what they do there doesn't make much sense, since they're using the current timestamp as the startDate and a timestamp in the future as endDate (and, yes, I also tried that).

[biemster]

In any case what they do there doesn't make much sense, since they're using the current timestamp as the startDate and a timestamp in the future as endDate (and, yes, I also tried that).

Haha well spotted, that does not make much sense indeed..

[dakhnod]

Please don't forget that apple is using 1.1.2001 as second 0 for some timestamps in the AirTags

[olivluca]

Yes, that's taken into account by using 978307200 at various places in request_reports.py, but that code in OpenHaystack is using timeIntervalSince1970 (due to the results and the fact that they inverted startDate and endDate I wouldn't trust it too much).

[dakhnod]

Is there something like a limit parameter? So something like limit=1 only gives one report max?

Also, is there a limit to how many hashes we can request per request?

[biemster]

I don't know of a limit parameter or anything besides what the OpenHaystack people found.
And I seem to remember there was a hash count limit of 256 but I can't find the source of that anymore. But this is easy to check of course with the request_reports script, I might do that once.

[davesenior9]

Thanks mate. Apologies for the double up.

[biemster]

The startDate and endDate now seem to be interpreted by the apple servers, I need to do a bit of follow up research if it now works as expected.

[dakhnod]

@biemster damn, that's big. I'll test it out.

[biemster]

Allright I jumped the gun here a bit. I noticed that "startDate": 0, "endDate": 0 gave me only the last 20 reports per tag now, while it used to give me everything they had. So I started typing two new dates to give me 10 days of reports, and that worked! All reports since August 8!
So without checking I assumed that that's how it works now, but in a more truer sense it just gives me all those reports, no matter what dates I give. As long as it is not two zeros..
sorry for the noise @dakhnod , I hope I did not get you too excited.

[smierd]

Has anyone made progress on this? Right now with only 3 tags I'm getting around ~3000 reports per query when I'm only searching for a few. I'm worried about requesting too much and getting blocked. Has anyone had issues with too many queries?

[Itheras]

Has anyone made progress on this? Right now with only 3 tags I'm getting around ~3000 reports per query when I'm only searching for a few. I'm worried about requesting too much and getting blocked. Has anyone had issues with too many queries?

Just remove start and end date you will get the newest 20.

[aircable]

@biemster

So I started typing two new dates to give me 10 days of reports, and that worked!

I could not figure out what the new syntax for the start-endDate is you were trying. Can you please elaborate?
For now, I always get one week of data, not 10 days.

[biemster]

@aircable if I remember correctly I just tried what's already in the current code (maybe switched to seconds, or microseconds, or that weird Cocoa epoch), that's all. It was just a red herring, didn't work.