Feedback & Ideas

[nono303]

Hi @jonamat,
Many many Thx for this project! 🤩

I tested and used it few days with success but when I wanted to make some modifications - since I have never developed in Go - I rewrote it in a dirty PHP script with a class scrapper.
It completes my list of topo & overlay (see https://github.com/nono303/kmlsuperoverlay) for Google Earth.

Therefore, I’ll not be able to directly make a PR but here are some modifications / improvements and ideas that I have or will apply
Here is the result of an update example, to illustrate these changes

What i've done

1. ServerURL override; my go server was listening on 127.0.0.1:3000 and running behind an Apache reverseProxy with endpoints for all kml links like https://www.mydns.com/topos/wikiloc/…
2. Direct use of png icons https://sc.wklcdn.com/wikiloc/images/pictograms/ge/{1..144}.png rather than /cmd/get-icons/get-icons.go which transforms the svg in /web/static/icons/ (might answer Increase SVG conversion precision #1)
3. Send the wikiloc-earth-layer.kml stub on / request rather than redirecting to your website)
4. Select number of pages to load with a nbpage parameter (1: 24, 2: 48...)
5. Server-side caching of downloaded files (html & json) with TTL
6. Transformation of the Document track into a Folder with activity type icon in left menu
   a. Compact display in the left menu of Folder Placemark-Point & Placemark-LineString elements with <Snippet>
   ⚠️ maxline=0 attribute must be present for NetworkLink, although this is not necessary in static kml

7. For my personal use:
a. Changing displayed name by the technical criteria (elevation – difficulty – elevation + – distance) ... Wikiloc name are often dirty ^^
b. Description rewrite Compacted in a single html table
c. Color of the layout style depending on the difficulty (compliant with SAC notation based on the principle that 'skill' corresponds to T1 to T5 ratings) … but Wikiloc skill are often dirty too!

What's in progress

1. Implementation of filters fix shield (again) #4. Since the use of filters requires authentication and the Wikiloc login.do endpoint is protected by a captcha.
   I have not (yet) found a way to do it automatically (the https://github.com/ivecanski/scrape_my_wikiloc_stats require solving the captcha via selenium...)
   a. Therefore, I use the wikiloc SESSION cookie retrieved from my authenticated browser to use the filters.
   Its lifespan being short, I will dig into the point to try to find automation (note that builtin Google Earth browser doesn't allow script to resolve captcha: it's impossible to login on wikiloc and share cookie :()
   b. In the meantime, I plan to create a small html page where I define my criteria and this dynamically generates a wikiloc-earth-layer.kml stub with the parameters passed to/init which are then used by /update.
2. The kml <cookie> transmitting the list of ids in GET: we can quickly reach the limit of 2048 characters. To bypass this:
   a. Passing the cookie in POST but I didn't find anything https://developers.google.com/kml/documentation/kmlreference?csw=1&hl=fr#cookie
   b. Passing a session identifier (server) into a kml cookie and saving the ids displayed on the server side corresponding to the kml cookie session

So, I would be glad to keep this topic open and share with you what can be improved or added on this wonderful tool - independently of the backend language :)

[jonamat]

Woah you've done a ton of work! Love your superoverlay project! I'm so glad that someone is interested in this project, I still use it when I go for a walk, but I thought I was the only one confortable with this way of finding new trails, so I never got around to fixing bugs or adding features. Thanks for the hints! Let me share my thoughts line by line:

1. Elegant, I like it!
2. I didn't know about those files, I will definitely use them too!
3. Yes I initially developed the tool for myself, and I didn't spend much time making it fit for others
4. Great idea! I need to figure out how users can set those parameters. I remember trying to make an interactive KML to send commands to the server, but no luck. maybe a settings page served from a specific url?
5. I was thinking the same, but I liked the stateless approach. At that time I didnt want to implement all the stack to manage a db for the cached trails (potentially turned into a Wikiloc replica :p).
6. I'm not super skilled with KMLs, I'll take a deeper look at what you've done!
   7c. That's really clever, I'll think about it

And about your in-progrss works:

1. Quick thought: have you ever considered authenticating through google oauth? it wont ask you for the captcha and should give you a session token, but not sure if it's possible
2. I also struggled with the limitations of the KML format. the only way I found to get around it is to re-process the cookies on the server, keeping the most recent data, adding the newer ones to the payload, and removing the oldest. I'd love it if Earth Pro could handle some client-side operations, it would be a game-changer for us

Let's keep in touch, I'll go back on this as soon as I can!

[nono303]

I think I'm done digging on my PHP wikiloc-earth-layer (more than enough for my personal use ⛰️)

Here are my PHP sources for inspiration (understandable procedures but not directly functional because requires functions from my internal framework)
📁 php_wikiloc-earth-layer.zip

• /WikilocScraper.php : the main class
  -/wikiloc.php: The Endpoint Handler
  -/wikiloc/*.php : les endpoints

Here's where I am

I've looked for different methods for automatic Wikiloc authentication on the server side, but none of them came up because they were unsuccessful or too complex / time-consuming.
The copy-paste of the cookies suits me perfectly
I just implemented the update of it on set-cookie value response which allows you to keep the session for 1 week a priori

Anyway - kml remains purely static
⏩ Here is the preliminary workflow implemented:

1. / : static kml with an html link: passive & un-checkable (see wikiloc_root.kml)
2. /form: a splendid HTML form in frenglish ^^
   
   • Work both in GoogleEarth internal or external browser, according to client configuration.
   • Note that I've added an optional post filter to return only tracks whose maximum altitude is higher than the desired value.
   • Now all values (filter parameters and sent trails IDs remain server-side, just passing a PHPSESSID between kml <> server
3. No change as described above - /stub => /init => /update et /legend (png overlay)

❓ The last point I'd like to solve but don't know is to be able to "merge / update" wikiloc_stub.kml (from /stub) onto wikiloc_root.kml (from /) for not having 2 Wikiloc entries in GoogleEarth.

But I'm not very good at <updates> on <NetworkLinkControl> and I don't think it's possible on a <Folder> without <NetworkLink>
If you ever have any ideas, I'm all ears!

Well now... All that remains is to prepare the 2024 climbing season 😎

[jonamat]

Yes Google KML reference is quite criptic about the use of "" and I found some insights in other projects like this one.
Impressive work, well done! 😁
Now I'm too busy with another project (I'm trying to automate the devices in my campervan, super funny) but in the spring/summer I'll be back to steal some of your ideas 😜. I will keep you updated!