[Data Explorer] Serialize redux state to share link

[anuveyatsu]

Moved from gitlab, originally created by @starsinmypockets

As a user I want to share a link the data explorer in the state it is currently in so I can share insight and have fun exploring data with friends

Accptance

• Update application state updates share url
• Share url instantiates application in correct state
• Data explorer with share links working deployed to Montreal

Tasks

• Serialize redux state to url -- use state reducer and add to state
  • Create widgets
  • Remove data (data should be refetched)
  • Capture filter settings:
    • ChartBuilder
    • MapBuilder
    • Filter Queries
• Build simple share / embed component
• Data Explorer endpoint in theme

Analysis

(from hackmd w/ Rufus + Anuar 2019-09-02)

Data Explorer can take and run ...

view-embed?redux= .....

Boot a data explorer and load that redux ...

/core/xyz ... looking at an explorer

share https://mysite.com/view-explorer?redux=...

iframe src=/view-explorer?redux=....&embed=true

1. I go to showcase page and play with one of the xplorerers

2. I take the link now in the share field which is /view-explorer?redux=xxx

3. I paste this in my browser

4. /view-explorer?redux=xxx

   • Get the redux state out of the url (url decode it)
   • Instantiate Data Explorer and set the redux state

ACTIONS

• We don't need share or embed links in frontend-v2
• They go in data explorer now ..
• Next is serialize redux state in and out ...
  • Serialize redux (remove _values on any resources)
  • Update share link / iframe link on changes
• Unserialize and use in /view-explorer

explorer-redux = {
  explorer_config_version: 
  datapackage: {},
  widgets: []
  // maybe some extras
}

[anuveyatsu]

Comment from Paul Walker:

A bit of further analysis.

Unloading widgets -> datapackage

Pros:

• should / could be good library code (unloadDatapackage?)

I'm guessing that this should have an interface as follows:

import { unloadDatapackage } from 'data.js'
import { serializeState } from './utils'

// ... reducer code
const unloadedWidgets = getState().widgets.map(widget => {A
  const newWidget = deepClone(widget)
  newWidget.datapackage = unloadDatapackage(newWidget.datapackage)
  return newWidget 
})

// add unloaded to state, etc

Questions:

1. In a populated datapackage (data attached) can we remove all data and _values attributes from all embedded resources and still have a valid (ie fetch-able datapackage)?
2. Maybe this would break data-inline type datapackage resources, in which case we need an exception for that type?

Cons Complicated

Save initial config on store

Save the initial config and use it to reload app, along with controls which could be updated.

So an application could be rehydrated with two attributes and one function call:

• config.WidgetsOrViews (immutable never changes)
• config.Controls (current state of application)
• loadData() (go get the data)

The output of this would be the application state, including data

[anuveyatsu]

Comment from Paul Walker:

Most of this is done:

• Serialized state in redux app works
• Iframe page loads explorer from serialized state in URL (if it fits)
• Share link with serialized state works (hardocded origin for the moment)

There is a problem that the state is too long for the url which has a limit of 20000k
I think that we need some sort of key/val storage which works like this:

1. Explorer state is updated
2. Serialized state is stored to redis
3. Share link points to frontend endpoint that pulls state from redis and instantiates page

• frontend link would be: http://montreal.ckan.io/data-explorer/HASH
  • 1. HASH is a hash of the json string

I could do further research on redis solution. I'm sure there is a freemium hosted solution

Open to other suggestions :)

[anuveyatsu]

Comment from Paul Walker:

Also maybe we can use JSONC.pack which is url friendly zip... it's very ugly / unreadable / looks kind of sketchy (ie - it looks like obfuscated code which I would probably not paste into a url myself because I wouldn't trust it)

JSON:

{"widgets":[{"active":true,"name":"Map","datapackage":{"views":[{"id":0,"title":"Murales%20en%20format%20geojson","resources":["murales_en_format_geojson"],"specType":"map"}]},"loading":false}],"datapackage":{"resources":[{"path":"https://montreal.l3.ckan.io/en/dataset/587de1ea-817b-4e10-b387-83ec89f94e1b/resource/bc1e71e4-269a-4fa9-a278-ab6bfd80ec23/download/murales.geojson","pathType":"remote","name":"murales_en_format_geojson","format":"geojson","mediatype":"application/geo+json","mimetype":"application/json","cache_url":null,"hash":"","description":"Cet%20ensemble%20de%20données%20présente%20les%20murales%20financées%20par%20la%20Ville%20de%20Montréal%20depuis%202007.%20Les%20murales%20répertoriées%20sont%20celles%20financées%20par%20les%20programmes%20suivants:%20Graffitis%20et%20murales%20(2007-2015),%20Projet%20pilote%20d%27art%20mural%20(2014-2015)%20et%20Programme%20d%27art%20mural%20(2016-).\r\n\r\nVeuillez%20noter%20que%20les%20photographies%20des%20murales%20liées%20dans%20l%27ensemble%20de%20données%20ne%20sont%20pas%20soumises%20à%20la%20licence%20de%20données%20ouvertes%20Creatives%20Commons%204.0.%20Les%20œuvres%20sont%20protégées%20par%20les%20droits%20d%27auteur%20d%27usage.%20Pour%20toute%20utilisation,%20veuillez%20contacter%20les%20ayant%20droits%20afin%20d%27obtenir%20les%20licences%20nécessaires.","created":"2019-08-20T13:04:09.502529","datastore_active":false,"package_id":"587de1ea-817b-4e10-b387-83ec89f94e1b","cache_last_updated":null,"state":"active","mimetype_inner":null,"archiver":{"is_broken_printable":"Downloaded%20OK","updated":"2019-08-20T13:04:15.236769","cache_filepath":"/tmp/archive/bc/bc1e71e4-269a-4fa9-a278-ab6bfd80ec23/murales.geojson","last_success":"2019-08-20T13:04:15.236769","size":126010,"is_broken":false,"failure_count":0,"etag":"\"7575193e1c3c66dfd425bf61e5f46787\"","status":"Archived%20successfully","url_redirected_to":"https://cc-p-minio.ckan.io/ckan/montreal/resources/bc1e71e4-269a-4fa9-a278-ab6bfd80ec23/murales.geojson?AWSAccessKeyId=2effdd1004072cb9&Expires=1566306315&Signature=eKO3AI%2BGPGUaRtMApJNZ%2Fflma4M%3D","hash":"d023a02c1f1646fcaded3a8075eb4059accf1088","status_id":0,"reason":"","last_modified":"2019-08-20T13:04:10","resource_timestamp":"2019-08-20T13:04:09.461130","mimetype":"application/json","cache_url":"https://montreal.l3.ckan.io/bc/bc1e71e4-269a-4fa9-a278-ab6bfd80ec23/murales.geojson","created":"2019-08-20T13:04:15.258228","first_failure":null},"last_modified":"2019-08-20T13:04:09.440232","position":0,"revision_id":"e82aa157-d0bd-48f4-91f1-8135ba972fec","url_type":"upload","id":"bc1e71e4-269a-4fa9-a278-ab6bfd80ec23","resource_type":null,"size":null,"title":"Murales%20en%20format%20geojson","encoding":"utf-8"}],"controls":{"showChartBuilder":false,"showMapBuilder":false}},"datastoreFilters":{}}

"㞂⃮ॠ☎悦Ű㎈׀浐Ⴠ예聮熠聎ʹ쀍ࠁ\ud960ⶱꢀⲖ8荔堩홙Bᰰ⠲蚆考ർࠀ\udb33湔陵䠂适恗ᚡ蔀찃\uda90柌ﱋ¬鉝귄⤸⺴鴈̖躗耾覨喍鼨䎳ꮈ.贒ܜ฀઀➚娠月/ꉡ䴪ꖖᐄⴌᨹ躒ᰱ伟�エꨨ➷꤯貦ࠇἀ֞䣂ȇሪ=ⳃ⮙ᰎ耝⪀㌪ﭖ⴪쐥곉곯㼣䈬쀫\u0007;ᐜ#ીⴵ쀑诀ଓ슋폡뎹밶耓鳁௺㴾댞ꖇ윅驼烏害쟢Ø⅘徽䈒᥮흒柗᧷ㅁ껆ᰁ荢璱腨苘⮥姄屮ᨰ섂㛊鹅舘荇ꄰ洮鰫䒋奬ࡘ鰉쀯煄㕹繂䚄슩ᩎܕ䄁쇰ິ妃耍伔ᘄ⁌Ѕ꼥榵\udbf0깙鮽읩삌⴪贋䜢꥔㐑阉ر掸Ṉᰩȁ쐍롘a䐙蓈큢緔昇땵쭀ɝ磌᱒㵱ꭆ閙瑦䕈彄朳啶㡆｡䰵ⅶ끦\u001a萁㜆긮責▫杄룣造掣ʖ　撛䙞쫛䪐ზࠨ⡁饼ற譁ẟ의䧑苷ᵈ䬆ېᠦ౲䆷ȗ晁倳\u0007ᛑ챁\udf03᱋㏇釼\n㷖ᑇ鋠Ĩ꠳\u0000嘃ᰌ㱷鴬仈슁౛迶細ꕌ엃࣬䣂⎈틍衑⢐Ⰺ憫昵觬祎㞍쑞戵怀畈䔶虓咙츆\udc17\u000b쳅ꇨ렟G䨋뇼㠑麉ጸᆒ䌝㌩⥑ﱭߑ跘읕ᦋⳠ૊둣霅ਃ규蔖舊齦씴粑؇燼\u0003꧋떴䱟ॵ굨ۇ蠛쟴ⴼ筈茝ୋᰍ烇ᾕ恑༣ӳᴀ敀爀臩抎\ud9b6Ȑ竆̊✋㇎Ƞ搜伤끲ᩒ탆\udb9c艀䐸Ẫኑ挶苉빱\udded圌訰衭㵃튌ᰉ旁ꕣ《삲堺똛䚣଀ﱦ쬓栈ይ狒꿇啕돪僘꺔ꀼ鞉␔䩈색ḍ销偾③ᕙ⸅Ä렌ࡃ㕨遛폅ৎ숙螩呆衒ဖ킞䅠湻觥秞⾗Š弎룁ᰒᚅ慏鐵쀣傕䅍㕲΅㣠ࡵΌጔ龆镽㰔齱䋕Ʝꌶ㞍ጐ࿷൉툌䄀睐鏦̄ࢅ뚩썀D\ud90e䊠碘ꌀސƥ\udc49穝陀⡥ࣸ焞䝐∵晥煛鼒ៃ䣐睑蔴쐅預؃薘跻⢅䑰⋣ዄ煼傑黎剜钤끪嚗ꔙ晛酲逾孑ೊ煴⊂瀀爜⎡\ud861䂎醔昻踓\udc65₀瓦ᇣ씈ꔜ\uda41ⷫ擃ꠚ樄ީስ㯈໑䆡ထ༤剀宒Ḉ䰔硰ඇׅ焨ຑ\uda57ᙹ⹳ߣ윻蕽\udc01㏠獏 䄾聜斣﷋선猏겲㾵₪ᐢ砪觠딈랜扌榇ㅦ㽱砜ף씏ꥡ똰菘܌蟐薆롖⪀䑞իㄜǢᗈ酠ቆ䠩ᔡꑴ膓ꐖ膑ꖠ\u001f鈄u\f꤃ၒɀ㻎Ť\t─/ƃ耨䨁䁇股醾㠓\u0006䀁䐀އȆ䠐씑㟡뀔⸠\ud844䇆ꢈ́>蘮û⽡넠庌　ࡁ஑РŔ뀀ɐ䂬ሄ瀀Ԡä\u0016ꆀb䋘ᾊ샴ۂ目ᤲ꘼蔁ㆆ싀飝᎘䞋褀禃됞춤㈛㸏쁐需鏷⺓ථ⠊噳嚚쬕町曮Ӕ∬⪈㡥ꃷ뺡쑹輨\udc22♄г텸縃鳶烷จ\udc7aꍕ蠃Ḃぐ煈永㖥됻凑眐ؘ䖴戡匆揌䖁\ud98aꪇ塛ݠッ蚈硟ࠑ҈䒈\udaee⌛钊搬陇띹ᨌ嘸㖹ܾ৩煣跀〆ᙤ큁쩀㙞ߏ堦劂Å蛊\ud910ݥܐ쪎਍᣼㆜젌㮀ऀ贇䩼㰜�誵숬벇ƮƂ쁘ࢋ駰ꀯ울%䞅틞ȴ뢟ᡆ\udc03๡툺ು弣䉋乁౨ᴱh钻ꮰ覮戾롈ꛭ∶⧌⏖犃梫湐⍝䇤嘅藐ꦀᳪ軁ᡢ噼뼥芨㔏≍猆㉆ի鋀匳檘\ud921晌㞁❩ފ浐㤴꘠ֵ聶ǁ㭼寂餴湚燅떃肥㹰崙ఁਡ䈀"

JSONC compression isn't a definitive solution since we will still easily exceed the url limit in some cases - if the resource description is long, for instance

[anuveyatsu]

Comment from Rufus:

let's not do key value stuff in redis. For now we can see if we could keep it simple e.g. strip the data package down to essentials.

Longer terms we just store views explicitly i.e. login to store and share ... (that's a separate issue).