Preparations for v0.15.0 release (#913)

* Preparations for v0.15.0 release * removed warning about v0.14.0 and v0.14.1 * Comment formatting * Fixed date in Changelog As well as formatting and spelling Co-authored-by: Georgia Kokkinou <[email protected]> * Update openwpm/task_manager.py Co-authored-by: Georgia Kokkinou <[email protected]> * Updated VERSION file * Getting GCSFS to work again * Set black to the version * Updated JS dependecies Co-authored-by: Georgia Kokkinou <[email protected]>
openwpm · May 10, 2021 · 6a40b94 · 6a40b94
1 parent 7e126af
commit 6a40b94
Show file tree

Hide file tree

Showing 30 changed files with 1,247 additions and 3,508 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -4,7 +4,7 @@ repos:
     hooks:
     -   id: isort
   - repo: https://github.com/psf/black
-    rev: 20.8b1
+    rev: 21.5b0
     hooks:
       - id: black
         language_version: python3

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,24 @@
 # Changelog
 
+## v0.15.0 - 2021-05-03
+
+Updates OpenWPM to Firefox 88
+
+This release reenables the support for stateful crawling
+
+If you are unsure what this means please have a look at our documentation on
+[ReadTheDocs](https://openwpm.readthedocs.io/en/latest/Configuration.html#stateful-vs-stateless-crawls)
+
+Other things that happened since last release:
+
+- Restored Docker build (#871) - All of our releases are now available on Dockerhub again
+- OpenWPM now monitors speculative connections (#872)
+- We introduced sphinx and publish our documentation to RTD (#863, #894 and #900)
+- Combined log_directory and log_file to log_path (#911)
+- Fixed a bug in the socket code of the WebExtension  
+  Thanks @shashigharti for the excellent bug report! (#912)
+- Fixed data loss issue reported by @bkrumnow (#902)
+
 ## v0.14.0 - 2021-03-12
 
 Firefox 86.0.1

diff --git a/README.md b/README.md
@@ -3,12 +3,6 @@ OpenWPM
 [![Build Status](https://github.com/mozilla/openwpm/workflows/Tests%20and%20linting/badge.svg?branch=master)](https://github.com/mozilla/openwpm/actions?query=branch%3Amaster)
 [![OpenWPM Matrix Channel](https://img.shields.io/matrix/OpenWPM:mozilla.org?label=Join%20us%20on%20matrix&server_fqdn=mozilla.modular.im)](https://matrix.to/#/#OpenWPM:mozilla.org?via=mozilla.org) <!-- omit in toc -->
 =======
- :exclamation:  **WARNING: OpenWPM version 0.14.0 and 0.14.1 are loosing data!  
-We have noticed ourselves and heard from others that the two most recent versions of OpenWPM are loosing both structured and unstructured data.  
-Until this is resolved we recommend using OpenWPM v0.13.0.  
-If you also experienced data loss please file an issue to aid the investigation**
-
-
 
 OpenWPM is a web privacy measurement framework which makes it easy to
 collect data for privacy studies on a scale of thousands to millions

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-0.14.0
+0.15.0
diff --git a/crawler.py b/crawler.py
@@ -223,7 +223,7 @@ def callback(success: bool) -> None:
         reset=True,
         retry_number=retry_number,
         callback=callback,
-        site_rank=site_rank,
+        site_rank=int(site_rank),
     )
     command_sequence.get(sleep=DWELL_TIME, timeout=TIMEOUT)
     manager.execute_command_sequence(command_sequence)

diff --git a/demo.py b/demo.py
@@ -22,19 +22,19 @@
 browser_params = [BrowserParams(display_mode="headless") for _ in range(NUM_BROWSERS)]
 
 # Update browser configuration (use this for per-browser settings)
-for i in range(NUM_BROWSERS):
+for browser_param in browser_params:
     # Record HTTP Requests and Responses
-    browser_params[i].http_instrument = True
+    browser_param.http_instrument = True
     # Record cookie changes
-    browser_params[i].cookie_instrument = True
+    browser_param.cookie_instrument = True
     # Record Navigations
-    browser_params[i].navigation_instrument = True
+    browser_param.navigation_instrument = True
     # Record JS Web API calls
-    browser_params[i].js_instrument = True
+    browser_param.js_instrument = True
     # Record the callstack of all WebRequests made
-    browser_params[i].callstack_instrument = True
+    browser_param.callstack_instrument = True
     # Record DNS resolution
-    browser_params[i].dns_instrument = True
+    browser_param.dns_instrument = True
 
 # Update TaskManager configuration (use this for crawl-wide settings)
 manager_params.data_directory = Path("./datadir/")

diff --git a/docs/environment-rtd.yaml b/docs/environment-rtd.yaml
@@ -3,31 +3,31 @@ channels:
 - main
 dependencies:
 - beautifulsoup4=4.9.3
-- black=20.8b1
+- black=21.5b0
 - click=7.1.2
 - codecov=2.1.11
 - dill=0.3.3
-- gcsfs=0.8.0
+- gcsfs=2021.4.0
 - geckodriver=0.26.0
-- ipython=7.22.0
+- ipython=7.23.1
 - leveldb=1.23
 - multiprocess=0.70.11.1
 - mypy=0.812
 - nodejs=14.15.4
 - pandas=1.2.4
 - pillow=8.1.2
-- pip=21.0.1
+- pip=21.1.1
 - pre-commit=2.12.1
 - psutil=5.8.0
-- pyarrow=3.0.0
+- pyarrow=4.0.0
 - pytest-asyncio=0.14.0
 - pytest-cov=2.11.1
-- pytest=6.2.3
+- pytest=6.2.4
 - python=3.9.2
 - pyvirtualdisplay=2.1
 - recommonmark=0.7.1
 - redis-py=3.5.3
-- s3fs=0.6.0
+- s3fs=2021.4.0
 - selenium=3.141.0
 - sentry-sdk=0.20.3
 - sphinx-markdown-tables=0.0.15
@@ -36,7 +36,7 @@ dependencies:
 - tblib=1.7.0
 - wget=1.20.1
 - pip:
-  - dataclasses-json==0.5.2
+  - dataclasses-json==0.5.3
   - domain-utils==0.7.1
   - jsonschema==3.2.0
   - plyvel==1.3.0

diff --git a/environment.yaml b/environment.yaml
@@ -3,31 +3,31 @@ channels:
 - main
 dependencies:
 - beautifulsoup4=4.9.3
-- black=20.8b1
+- black=21.5b0
 - click=7.1.2
 - codecov=2.1.11
 - dill=0.3.3
-- gcsfs=0.8.0
+- gcsfs=2021.4.0
 - geckodriver=0.29.0
-- ipython=7.22.0
+- ipython=7.23.1
 - leveldb=1.23
 - multiprocess=0.70.11.1
 - mypy=0.812
 - nodejs=14.15.4
 - pandas=1.2.4
 - pillow=8.1.2
-- pip=21.0.1
+- pip=21.1.1
 - pre-commit=2.12.1
 - psutil=5.8.0
-- pyarrow=3.0.0
+- pyarrow=4.0.0
 - pytest-asyncio=0.14.0
 - pytest-cov=2.11.1
-- pytest=6.2.3
+- pytest=6.2.4
 - python=3.9.2
 - pyvirtualdisplay=2.1
 - recommonmark=0.7.1
 - redis-py=3.5.3
-- s3fs=0.6.0
+- s3fs=2021.4.0
 - selenium=3.141.0
 - sentry-sdk=0.20.3
 - sphinx-markdown-tables=0.0.15
@@ -36,7 +36,7 @@ dependencies:
 - tblib=1.7.0
 - wget=1.20.1
 - pip:
-  - dataclasses-json==0.5.2
+  - dataclasses-json==0.5.3
   - domain-utils==0.7.1
   - jsonschema==3.2.0
   - plyvel==1.3.0