🚀 Release v1.78.0

[YuryHrytsuk]

Release version

1.78.0

Commit SHA

d8f172080da94d6f57e77af3e5a30f8fb4a8be5a

Previous pre-release

https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_

Planned date

07.11.2024

Did the commit CI suceeded?

• The commit CI succeeded.

Motivation

• Regular sprint release

Changes

• 🐛 fix version control doesn't populate projects_to_products table #6357 by @matusdrobuliak66
• 🎨 [Frontend] TIP lite: add teaser button #6359 by @odeimaiz
• 🎨E2E: adapt test to new service startup status #6363 by @sanderegg
• 🎨🐛 [Frontend] Folders tree: improvements #6360 by @odeimaiz
• 🎨🐛 [Frontend] Folders tree: improvements II #6364 by @odeimaiz
• 🐛 do not allow moving folder to a child folder #6370 by @matusdrobuliak66
• 🎨 [Frontend] Prettify markdown's outcome #6369 by @odeimaiz
• 🐛 Fixed Redis issue where keys ended up in the wrong deployment. #6368 by @GitHK
• ♻️CI: increase timeout on autoscaling to fix mypy slowness #6375 by @sanderegg
• 🐛 [Frontend] Fix Automatic Shutdown = 0 #6372 by @odeimaiz
• ✨ Add: Tracing via opentelemetry #6168 by @mrnicegyu11
• 🎨 [Frontend] TIP lite: new logos #6377 by @odeimaiz
• 🐛 Wrong opentelemetry-collector config filename #6382 by @mrnicegyu11
• 🎨 [Frontend] Workspaces and Folders: Sync tree #6373 by @odeimaiz
• 🎨Redis: add optional Redis username #6381 by @sanderegg
• 🐛 fix filtering when project shared with product groups #6379 by @matusdrobuliak66
• 🐛 rabbitmq queue name prefix is unique during the lifecycle of the container #6365 by @GitHK
• ✨Adds metrics regarding new style dynamic services timings #6347 by @GitHK
• 🎨 [Frontend] Make the frontend dont-show-folders compatible #6387 by @odeimaiz
• ✨[Frontend] Denylist for (lite) registration emails #6385 by @odeimaiz
• 🎨 backend pulls inputs before iframe is displayed #6392 by @GitHK
• 🚨🐛 Mitigate dy-service file upload bug: Increase traefik timeouts #6395 by @mrnicegyu11
• ✨ [Frontend] Force reload no-cache #6397 by @odeimaiz
• 🐛⚗️Prometheus instrumentation incorrectly setup #6398 by @sanderegg
• ✨E2E: increase timeout for video streaming and improve a bit logs about websockets #6384 by @sanderegg
• 🎨Clusters keeper/use ssm (🚨change in private clusters) #6361 by @sanderegg
• ♻️CI: autoscaling mypy takes longer and longer. #6417 by @sanderegg
• 🐛 Improved Error Handling for Missing Billing Details #6418 by @pcrespov
• 🎨 Enhances Product parsing to strip whitespaces in host_regex #6419 by @pcrespov
• 🐛 [Frontend] Fix: Share and Leave Study #6401 by @odeimaiz
• ✨ [Frontend] Workspaces & Folders: Merge Workspace header and Folder breadcrumbs #6414 by @odeimaiz
• 🎨 [Frontend] Access Full TIP #6423 by @odeimaiz
• 🐛 [Frontend] Do not list/request /frontend/ service's pricing plans #6424 by @odeimaiz
• Bump actions/setup-node from 4.0.3 to 4.0.4 #6425 by @dependabot
• 🎨 Persistent logs when GC removes services #6403 by @GitHK
• 🎨 The user can remove themselves from the project/workspace. 🚨  #6415 by @matusdrobuliak66
• ♻️ Maintenance: removing old folders #6383 by @matusdrobuliak66
• 🎨 improve DB foreign key dependencies (🗃️) #6428 by @matusdrobuliak66
• ✨ dynamic-scheduler pushes service state to the frontend (⚠️ devops) #5892 by @GitHK
• 🐛 [Frontend] Fix: Do not listen to output related backend updates if the node is a frontend node #6434 by @odeimaiz
• ✅ e2e: new workflow to test tiplite #6388 by @pcrespov
• 🎨 [Frontend] App mode: Scrollable Instructions #6430 by @odeimaiz
• 🎨 publish port events to frontend #6396 by @GitHK
• 🔒️ Strip credentials form image VCS #6433 by @GitHK
• ⬆️ Sep tests and tooling upgrade #6431 by @GitHK
• ✅ e2e: fixes check on full version #6445 by @pcrespov
• 🐛 Fixes flaky tests in service-library test_deferred_tasks.py #6439 by @GitHK
• 🐛 [Frontend] Announcements: allow in ribbon only #6440 by @odeimaiz
• 🎨 improve error handling on pricing plans #6436 by @matusdrobuliak66
• 🐛[FrontEnd] Better error handling of invalid pre-registration data #6438 by @pcrespov
• ✅ e2e: fixes wrong sequence in classic tip  #6450 by @pcrespov
• 🎨 improve tip e2e test #6448 by @matusdrobuliak66
• 🐛 Fixes issue with parsing logs polluting sidecar logs #6442 by @GitHK
• 🎨 [Frontend] TIP: Rebranding #6446 by @odeimaiz
• Adding favicon for all devices #6447 by @jsaq007
• 🎨 [Frontend] Unify border-radius #6451 by @odeimaiz
• 🐛 fix tip e2e test #6458 by @matusdrobuliak66
• 🎨 adding folder_id to project resource #6460 by @matusdrobuliak66
• ✅ e2e: Final touches to tip testing #6459 by @pcrespov
• ✅ e2e: Fixes typo in tests #6468 by @pcrespov
• ✨ [Frontend] Keep View Mode in localStorage #6463 by @odeimaiz
• 🎨 [Frontend] Move multiple studies at once #6457 by @odeimaiz
• ✨ New repository helpers on asyncpg #6465 by @pcrespov
• ♻️ Preparations in webserver to integrate asyncpg engine #6466 by @pcrespov
• ✨ introduce /projects:search full search #6464 by @matusdrobuliak66
• 🎨 Frees up space on runner for [build] docker images part of the CI #6475 by @GitHK
• 🐛🎨 Fixes unhandled config error and new log helpers for better troubleshotting #6432 by @pcrespov
• 🎨 [Frontend] S4L and S4L lite: Include deprecated versions in plus button, they should all be updatable to a non deprecated one #6480 by @odeimaiz
• 🐛 [Frontend] Guided Tours: point to visible selector or skip #6481 by @odeimaiz
• 🔨📝 Misc improvements in tooling and doc #6478 by @pcrespov
• 🐛🎨 [Frontend] Workspaces & Folders: Improve context syncing #6482 by @odeimaiz
• 🎨 Handles wallet forbidden error and enhances handling of unexpected errors #6444 by @pcrespov
• 🔨 Doc and scripts to create read-only user in PostgreSQL #6426 by @pcrespov
• 🐛 [Frontend] Move grouped nodes #6486 by @odeimaiz
• 🎨 [Frontend] Handle LongRunningTask errors better #6485 by @odeimaiz
• 🐛 [Frontend] Fix for Folders disabled scenario #6488 by @odeimaiz
• 🐛 archiving_utils creates deterministic zip archives #6472 by @GitHK
• 🐛 Fixes duplicates in tags listings and new priority to enforce order  #6479 by @pcrespov
• ✨Releases public-api 0.7 #6489 by @pcrespov
• ♻️ Polished and modernised agent service ⚠️ #6452 by @GitHK
• 🎨 improve project full search #6483 by @matusdrobuliak66
• 🎨 efs improvements (group extra properties) 🗃️ #6493 by @matusdrobuliak66
• 🐛 Front-end: fixes PO center widget #6496 by @pcrespov
• 🎨 Adds authentication for new style dynamic services and platform vendor services ⚠️ #6484 by @GitHK
• 🎨 Expose env var to raise timeout of downloading/uploading data to and from S3 #6494 by @GitHK
• 🔨 New sql script template to remove readonly user #6500 by @pcrespov
• 🐛 Fixes bad formatting while handling missing configuration errors #6501 by @pcrespov
• ♻️ increase e2e tip test timeout #6504 by @matusdrobuliak66
• 🎨 EFS Guardian: adding size monitoring #6502 by @matusdrobuliak66
• 🐛 Fixes auth product error in vendor services 🚨 #6512 by @pcrespov
• ♻️🔨 Unexpected mypy upgrade revealed configuration and code failures #6527 by @pcrespov
• ♻️Maintenance: adjustments to Makefile to use UV instead of pyenv #6532 by @sanderegg
• 🎨 [Frontend] Search across all Workspaces and Folders #6470 by @odeimaiz
• ✨ [Frontend] Listen to the serviceStatus websocket event #6487 by @odeimaiz
• 🎨 [Frontend] Rebranding: S4L lite -> Sim4Life.lite #6524 by @odeimaiz
• 🐛 Fix aiohttp server autoinstrumentation #6391 by @mrnicegyu11
• 🐛 [Frontend] Fix replaceTokens function #6539 by @odeimaiz
• ⬆️♻️ Upgrades pint library as well as Dockerfile syntax to address deprecation or syntax warnings #6521 by @pcrespov
• Bump codecov/codecov-action from 4.5.0 to 4.6.0 #6497 by @dependabot
• 🐛 Remove Efs Guardian for devel model #6540 by @matusdrobuliak66
• ♻️ Renaming session cookie once again #6544 by @GitHK
• ✨ [Frontend] Listen to the stateInputPorts and stateOutputPorts websocket events #6538 by @odeimaiz
• ⬆️ ♻️ Upgrade pagination customisation in api-server #6545 by @pcrespov
• 🎨 Enhanced error handling and troubleshooting logs helpers #6531 by @pcrespov
• 🐛 [e2e, Frontend] Use replace instead of replaceAll #6549 by @odeimaiz
• 🐛🎨 [Frontend] Load folders after going back to the dashboard #6543 by @odeimaiz
• ♻️Sim4Life computational jobs are not parented correctly #6542 by @sanderegg
• 🎨 [Frontend] TIP.itis.swiss: Show Phase out message at Request Account #6552 by @odeimaiz
• 🐛🎨 [Frontend] Sim4Life.lite: Fix and add guided tours #6541 by @odeimaiz
• ♻️ Refactor DiskUsage functionality to support efs-guardian #6536 by @GitHK
• ♻️ Tests bad parsing of jobs custom metadata #6559 by @pcrespov
• 🐛 [Frontend] Reuse client_session_id from socket's query instead of creating a new one #6564 by @odeimaiz
• 🎨🐛 [Frontend] Retry /auth/logout call if times out #6558 by @odeimaiz
• ♻️ 📝 Minor refactor and doc of autoscaling service #6551 by @pcrespov
• 🐛 [Frontend] Guest viewers #6565 by @odeimaiz
• 🎨 [Frontend] Search with tags filter #6554 by @odeimaiz
• 🎨 increase timeout in tip e2e test #6568 by @matusdrobuliak66
• ♻️ aiohttp deprecation: Using web.json_response to return 2XX responses instead of raising HttpException #6563 by @pcrespov
• 🎨 [Frontend] Enh: Show pending service_message in loading page #6570 by @odeimaiz
• 🐛E2E: fixes how videostreaming checked #6569 by @sanderegg
• 🎨 EFS Guardian adding data removal background task #6562 by @matusdrobuliak66
• 🎨Dask sidecar: use reproducible zipfile library #6571 by @sanderegg
• 🐛Dynamic schldr: deferred tasks passing invalid kwargs parameters to exceptions #6573 by @sanderegg
• 🐛Deferred tasks: issue with library changing its syntax #6575 by @sanderegg
• ⬆️Fast depends: update version due to syntax breaking change #6577 by @sanderegg
• ♻️Maintenance: Improve UV usage #6566 by @sanderegg
• 🎨 [Frontend] Make the Workspace UI element more distinguishable #6576 by @odeimaiz
• 🎨 [Frontend] Small screens: Let them go #6390 by @odeimaiz
• 🐛 [Frontend] Fix: Download logs with newlines (Firefox) #6583 by @odeimaiz
• 🐛Maintenance: upgrade faststream dependency #6586 by @sanderegg
• 🐛Deferred tasks: set default log level of messages in Faststream broker to DEBUG #6589 by @sanderegg
• 🐛 EFS Guardian - not need of owner information if project lock in MAINTAINING state 🚨  #6581 by @matusdrobuliak66
• 🐛 Fix opentelemetry errors in autoscaling microservice #6582 by @mrnicegyu11
• 🐛 [Frontend] Fix: duplicated /folders call #6587 by @odeimaiz
• ♻️ [Frontend] Context property for the StudyBrowser #6598 by @odeimaiz
• 🐛 [Frontend] Fix: Avoid null parameters in requests #6601 by @odeimaiz
• 🔒️Fix codecov integration #6606 by @sanderegg
• 🔨Fix codecov2 #6607 by @sanderegg
• 🔨Fix codecov3 #6608 by @sanderegg
• 🐛 FIX: project patch when wallet not shared #6602 by @matusdrobuliak66
• ♻️ Remove PUT project endpoint #6604 by @matusdrobuliak66
• 🔨Fix codecov4 #6610 by @sanderegg
• 🎨 [S4L] New studies: Point to the upgraded key services #6611 by @odeimaiz
• 🐛 [Frontend] Initialize Study Browser container spacing in list mode #6613 by @odeimaiz
• ♻️Reduce noisyness of gunicorn and socketio in webserver logs #6616 by @sanderegg
• 🐛 [Frontend] Fix: allow moving to root folder #6615 by @odeimaiz
• 🎨 [e2e-frontend] Navigation Bar and User Menu #6603 by @odeimaiz
• ✨ Trash projects #6579 by @pcrespov
• 🔨Use one cache for UV and pin version to 0.4 #6622 by @sanderegg
• use per package cache #6623 by @sanderegg
• 🔨UV cache: fix typos #6624 by @sanderegg
• Bump actions/setup-node from 4.0.4 to 4.1.0 #6618 by @dependabot
• 🔨 Code coverage: fix api code generation + cleanup #6625 by @sanderegg
• ♻️ dynamic-sidecar rpc interfce namespace is now tied to the node_id #6614 by @GitHK
• 🐛 [Frontend] Fix probe #6620 by @odeimaiz
• 🐛 removes unsupported parameter #6612 by @GitHK
• 🎨 [Frontend] Open Study location from Search context #6630 by @odeimaiz
• 🎨 [Frontend] Publish template UI/UX #6617 by @odeimaiz
• 🔨CI: disable uv caching #6636 by @sanderegg
• ♻️ rearranging webserver tests #6633 by @matusdrobuliak66
• 🎨 notify frontend about current efs disk space #6520 by @matusdrobuliak66
• 🎨 [e2e-frontend] Check study browser #6635 by @odeimaiz
• 🐛 Fixed issue with accumulating tracked services #6631 by @GitHK
• 🐛 [Frontend] Service Browser and Service Catalog use the same function #6638 by @odeimaiz
• 🎨 [Frontend] Show EFS data storage #6639 by @odeimaiz
• 🎨 introduce logger filtering with ENV var ⚠️ #6596 by @matusdrobuliak66
• 🎨🐛 [Frontend] Reduce the initial number of patch calls #6641 by @odeimaiz
• 🐛 [Frontend] Fix: Selected Pricing Unit bgColor #6646 by @odeimaiz
• ♻️ Changing == by is_ in sqlalchemy queries #6654 by @pcrespov
• 🐛 [Frontend] Fix Service browser #6659 by @odeimaiz
• ♻️ introduce webserver 4 tests #6663 by @matusdrobuliak66
• ♻️ Refactor RUT (renaming/rearranging/error handling) #6648 by @matusdrobuliak66
• ♻️Director-v2: refactor computational scheduler #6652 by @sanderegg
• ♻️ Mark new trash web-api as dev-features #6665 by @pcrespov
• 🎨 [e2e-frontend] Service browser #6664 by @odeimaiz
• 🐛 Avoids raising error when composing disk usage #6660 by @GitHK

No response

Devops check 👷

• Merge EFS https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1020
• Check than new AUTH Logic with cookies works for users (log out may be required) MD / YH
• Merge Remove redis commander docker node label
• Merge Remove jaeger docker node label
• Merge and apply sim4life private static ec2
  • Make sure computational clusters will not be broken after this switch. Consult with @sanderegg
  • Enable S3 VPC Endpoint?
• tip.science: update autoscaling config (sync with @sanderegg)
  • Remove old key
• tip.science: use aws redis
  • update creds
• nih prod: use aws redis
  • update creds
• INFO Log Level from Simcore services
• Fix postgres route on inhouse tip
  • Merge [v1.78.0] PROD inhouse tip: fix simcore traefik postgres route osparc-ops-environments#846
  • Verify it works (e.g. via adminer)
• simcore datcore-adapter service periodically fails to connect otel-collector host in inhouse-master deploy osparc-ops-environments#825
• sim4life: enable AWS EFS
• nih prod: enable AWS EFS
• Update SSL Certificates
  • TIP inhouse
  • Dalco
  • NIH PROD
  • Sim4life PROD
• Rotate traefik password on in-house TIP
• Add Vendors Stack to CI
• Unify ssh key names osparc-ops-environments#861

Tests assessment: e2e testing check 🧪

No response

Test assessment: targeted-testing 🔍️

No response

Test assessment: user-testing 🧐

No response

Summary 📝

• Prepare release link

make release-prod version=v1.78.0  git_sha=<commit_sha>

• Draft release changelog
• Announce maintenance ( ** ANNOUNCE AT LEAST 24 HOURS BEFORE ** )
• redis add an entry in db scheduled_maintenance with key maintenance {"start": "2023-03-06T13:00:00.000Z", "end": "2023-03-06T15:00:00.000Z", "reason": "Release <vX.X.0>"}. Note the time is in UTC, please adjust accordingly
  • sim4life.io
  • osparc.speag.com
  • osparc.io
  • tip.science
• status page (https://manage.statuspage.io/)
  • sim4life.io
  • s4l-lite.io
  • osparc.io
  • tip.science
• mattermost channels
  • maintenance
  • power users
• Publish draft draft
• Check release CI

Releasing 🚀

• Maintenance page up (https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/pipeline_schedules)
• Check hanging sidecars. Helper command to run in director-v2 CLI simcore-service-director-v2 close-and-save-service <uuid>
• Remove redis key
• Check deployed
  • sim4life.io
  • osparc.speag.com
  • osparc.io
  • tip.science
• Check testing endpoint ex. https://testing.osparc.speag.com/
• Update status page
• Update Release notes if hotfix (https://github.com/ITISFoundation/osparc-issues/blob/master/scripts/run_after_hotfix_to_prod.py)
• Check e2e runs
• Announce in mattermost

:tada: https://github.com/ITISFoundation/osparc-simcore/releases/tag/v<M.m.0>

[GitHK]

•  MERGE THIS before releasing

[sanderegg]

Update autoscaling bash scripts to have the correct GRAYLOG_URL setup in all the production deployment BEFORE releasing.

MR: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1045

[matusdrobuliak66]

OPS:

Add EFS to osparc.io and sim4life.io AWS deployments

DEV

• Run e2e against deploy to check for 🐛 Fixes auth product error in vendor services 🚨 #6512
• @mguidon @sanderegg @matusdrobuliak66 Double-check that the parent folder ID is correctly populated after running isolver jobs from sim4life
  • @matusdrobuliak66 please check that issue: Parent project and node are wrong in resource_tracker_service_runs table #6530 the problem was identified as s4l-based. the release of the new version should fix it. Please check it out.
• @mguidon @matusdrobuliak66 Double check pricing plans for newly published services

[jsaq007]

👍🏼

[YuryHrytsuk]

Release issues

sim4life.io:

• aws lb target group names are terraform id (unreadable)
• docker login required on simcore nodes (manually fixed)
• python3.11 is not install on machines (update ansible scripts?)

tip.science:

• broken s3 endpoint (fixed)

ops:

• deploy_ops stage shall only be run manually (in STAG/PROD)
• REGISTRY_S3_ENDPOINT and S3_ENDPOINT usage is mixed up in osparc-ops-environments repo
• AWS TIP is the only deployment in non-default region. Too late to catch bugs
• make release-prod is tedious to repeat for all deployments
• external maintenance test is broken?

Follow up ITISFoundation/osparc-ops-environments#867

[sanderegg]

Release issues

all deployments:

• the _TRACING environment variables are missing for all services but the webserver, needed to add them on the fly.
• @GitHK is creating a PR to add them in simcore,
• @mrnicegyu11 they are missing in all staging deployment and likely in master deployments
• ENV variables are not straightforward to use osparc-ops-environments#866

[matusdrobuliak66]

Update:

• hotfix branch hotfix_v1_78_x was created
  • cherry pick: e11f17c
• As there were issues with the CI timing out a couple of times, I built it locally and pushed it to DockerHub. I have restarted director-v2 with this tag: itisfoundation/director-v2:matus-hotfix-v1.78.x-e11f