Bug during import when QueuedTracking enabled

[HubertNodzak]

Hello,
PIWIK: 3.2.1
QueuesTracking: 3.1.0

During import json files from Apache (when above plugins is enabled), there is an error:
2017-12-07 08:13:09,741: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,741: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:09,872: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,872: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:09,956: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,956: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,093: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,093: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,180: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,180: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,263: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,263: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,403: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,403: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,537: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,538: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,668: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,668: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,753: [INFO] bulk tracking returned invalid JSON

OS Centos 7 (on both systems).

[sgiehl]

Do you still have this issue? If so would you mind to share the command you used to import the log files?
We would aim to fix this if possible to reproduce

[jsandner]

I experienced this error too on some Log-Lines. I am using format "common_complete" (Apache vhost_combined). The error occurs when:

• Method is HEAD instead of GET
• UserAgent is "-"
  (see During log importing with queuedtracking enabled bulk tracking returned invalid JSON plugin-QueuedTracking#34)

[nomandera]

I am consistently seeing this error.

There are several tickets about this issue but none clarify if this is a warning or an error. Specifically are the logs still bulk imported correctly?

I have tested a number of things so far:

• Local import using Debian and official apt package
• Remote import using windows and official python
• Remote import using windows and cygwin
• Local and remote single file and wildcard multiple file import
• Logs from four differernt IIS 7 cluster nodes
• Logs from three different sites
• With --recorders=1 and --recorders=2
• With and without --token-auth=
• With and without --enable-http-errors
• With and without --enable-http-redirects
• With and without --enable-static
• With and without --enable-bots
• With and without --accept-invalid-ssl-certificate

None of the above makes any difference at all.

This setup is four clustered front end boxes, one back end box for private access, one redis server and one sql server

• Matomo version: 3.7.0 (official Deb)
• MySQL version: 10.1.37-MariaDB-0+deb9u1
• PHP version: 7.0.33-0+deb9u1

All the standard addons plus

• QueuedTracking
• HeatmapSessionRecording
• MediaAnalytics
• BulkTracking (default but just listing for clarity)

[tsteur]

I wonder if it would be helpful to have a flag in log importer like --ignore-queued-tracking which appends a queuedtracking=0 to the tracking request and tells Matomo to avoid the queued tracking but instead execute them directly. This queuedtracking=0 would be also helpful when directly wanting to test some tracker issues etc.

When using log importer, there's usually not much benefit (or none, it actually adds some overhead) of using queued tracking so that can be useful. This way the output would be correctly.

Queued tracking can currently not really return a valid response because it wouldn't know whether any tracking requests were imported correctly or not. Maybe we would simply always need to append &queuedtracking=0 when importing data.

[tsteur]

created matomo-org/matomo-log-analytics#240 and matomo-org/plugin-QueuedTracking#96

[nomandera]

Just to follow up on this. I cannot test on the specific problem environment this now as the official Matomo apt package is still at 3.7.0 which sticks QueuedTracking at 3.3.1

Please update Piwik 3.7.0 to a newer version, Piwik >=3.8.0-rc2,<4.0.0-b1 is required.

Since this is a production environment I am limited by policy to manually bypass/update.

When the updates appear I will retest.

[tsteur]

I think this issue can be closed? What is left todo is matomo-org/matomo-log-analytics#240 and some docs

[nomandera]

Coincidentally I was working on this in the last few days.

I have tried everything I can, using all the latest advice and code, to stop the gif being returned rather than valid JSON. With all the recent changes the import actually works correctly albeit the weird gif error still persists.

I dont know if it is something specific to my setup but at least now it works.

[tsteur]

@mattab see #12340 (comment) I suppose we can close the issue as only matomo-org/matomo-log-analytics#240 is left?