Merge balazs-patchset #13

balazsgaspar · 2021-04-19T20:19:24Z

Fixes #8, #9, #11 and #12
Tested against latest cdp716 stack stack.cdp716.sh

instance ip address WEB CM CEM NIFI NREG SREG SMM HUE CDSW Model Status Viz Status
aws_instance.web[null] 3.127.215.88 Ok
aws_instance.cluster[0] 18.196.81.128 Ok Ok Ok Ok Ok Ok Ok Ok deployed running
aws_instance.cluster[1] 18.184.166.69 Ok Ok Ok Ok Ok Ok Ok Ok deployed running

Fixed a typo ("yum" -> "git")

Merge upstream changes

…e2ai-workshop

…ge2ai-workshop

Change Lab 6 from Hue to CDP Data Visualization

…in cdsw setup script.

Updated Lab 6 instructions for CDV

Fixing Kudu timestamp column issue

asdaraujo · 2021-05-21T05:03:21Z

Thanks a lot, @balazsgaspar ! This is great!
Could you please take a look at the items I've commented on?

Thanks again!

asdaraujo · 2021-05-21T05:05:28Z

setup/terraform/resources/cdsw_setup.py

-    for project in r.json():
-        if project['name'] == project_name:
-            project_id = project['id']
+


Any particular reason for removing the try-except block?
I added that to help with troubleshooting. Without it the HTTP messages are not show properly and it makes it more difficult to find the root cause of failures in certain cases.

asdaraujo · 2021-05-21T05:15:35Z

setup/terraform/resources/cdsw_setup.py

+    r = s.post(CDSW_API + '/users/admin/projects', json={'template': 'git',
+                                                         'project_visibility': 'private',
+                                                         'name': project_name,
+                                                         'gitUrl': 'https://github.com/balazsgaspar/edge2ai-workshop'})


replace balazsgaspar with cloudera-labs

asdaraujo · 2021-05-21T05:26:38Z

setup/terraform/resources/cdsw_setup.py

+models = [m for m in r.json() if m['name'] == model_name]
+if models and models[0]['latestModelDeployment']['status'] == 'deployed':
+    print('Model is already deployed!! Skipping.')
+    #exit(0)


In case the model has already been deployed, we don't want to re-run the steps below either (Add engine, variables, etc). At this point we could jump directly to the beginning of the VizApps section.

It would be better to encapsulate the model deployment and the vizapps deployment in separate functions and call these functions from the main script section.

This exit(0) would turn into a return, so that the whole deploy_model function would terminate and the deploy_vizapps one would be executed.

asdaraujo · 2021-05-21T05:29:47Z

setup/terraform/resources/cdsw_setup.py

+print('Viz project ID: %s'% (viz_project_id,))
+
+print('# Add custom engine for Data Visualization server')
+#docker.repository.cloudera.com/cloudera/cdv/cdswdataviz:6.2.3-b18


I don't think this comment is necessary

Done. Need to test with a more recent Viz image and if everything works, will update the image tag in the script as well.

asdaraujo · 2021-05-21T05:30:17Z

setup/terraform/resources/cdsw_setup.py

+print('Engine Image ID: %s'% (engine_image_id,))
+
+print('# Set new engine image as default for the viz project')
+#docker.repository.cloudera.com/cloudera/cdv/cdswdataviz:6.2.3-b18


asdaraujo · 2021-05-21T05:31:06Z

setup/terraform/resources/cdsw_setup.py

+print('Project image default engine Image ID set to: %s'% (project_engine_image_id))  
+
+print('# Create application with Data Visualization server')
+# https://github.infra.cloudera.com/Sense/cloudera-sense/blob/master/services/web/server/api/v1/controllers/applications/partial.swagger.yaml


what's this URL for?

asdaraujo

I like the idea of the "lite" versions of the workshops. However, maintaining content is not an easy task when there are changes/updates to the product.

Having a duplicate of every lab with a few changes here and there will make it more difficult to keep them up-to-date and they will end up diverging over time.

Do you think it would be possible to have a single document with optional steps that could be skipped for a "lite" lab? That would make it a lot easier to maintain.

asdaraujo · 2021-05-21T06:14:11Z

setup/terraform/resources/cdsw_setup.py

+    if app_status == 'running':
+      print('# Viz server app is running. CDSW setup complete!')
+      break
+    elif build_status == 'stopped':


this should be app_status instead of build_status

asdaraujo · 2021-05-21T06:14:22Z

setup/terraform/resources/cdsw_setup.py

+    elif build_status == 'stopped':
+        # Additional error handling - if the app exists and is stopped, start it?
+        break
+    elif build_status == 'failed':


asdaraujo · 2021-05-21T06:17:47Z

setup/terraform/resources/cdsw_setup.py

+print('Set unauthenticated access flag to: %s'% (r.json()["allow_unauthenticated_access_to_app"],))
+
+print('# Add project for Data Visualization server')
+project_name = 'viz'


Can we give it a nicer name so that it looks better on CDSW? ;) E.g. "VizApps Workshop"

asdaraujo · 2021-05-21T06:27:55Z

streaming_lite.adoc

+
+image::images/hue_timestamp_fixed.png[width=800]
+
+. We will now create a simple interactive real-time dashboard. Open CDP Data Visualization and log in. Select the Data tab and click NEW CONNECTION.


It might be worth explaining how to "Open CDP Data Visualization and log in", through CDSW. In real life they won't have the handy web server link to help.

The first time I had to find a VizApps link in CDSW it probably took me 1 hour to figure out :)

Also, log in with which credentials?
I found that admin/supersecret1 doesn't work. We could document vizapps_admin/vizapps_admin, but I rather figure out how to use admin/supersecret1, so that's consistent with everything else.

asdaraujo · 2021-05-21T06:30:04Z

setup/terraform/resources/cdsw_setup.py

+    'description': 'viz server app',
+    'kernel': 'python3',
+    'memory': 2,
+    'name': 'viz-server',


Nicer name for display? :) <Something> Application, so it's more obvious in CDSW

asdaraujo · 2021-05-21T07:23:36Z

streaming_lite.adoc

+
+When both tabs have been filled, test the connection. You should see "Connection Verified". Hit Connect.
+
+. We can explore the databases and tables of the newly added connection. From the newly created Impala connection open the default database and select "sensors". A preview with sample data will be loaded.


Can you please mention the "Connection Explorer" tab?

asdaraujo · 2021-05-24T01:02:52Z

sql_stream_builder_lite.adoc

+. On the Cloudera Manager console, click on the Cloudera logo at the top-left corner to ensure you are at the home page and then click on the *SQL Stream Builder* service.
+
+. Click on the *SQLStreamBuilder Console* link to open the SSB UI.
+.. On the first access you will be prompted to accept the use of cookies on this page. Click *Accept* to dismiss that message.


This is an example of what I mentioned of it being harder to maintain multiple similar documents. This prompt does not exist in the latest version any longer. I had already updated the original lab to remove it, but the copy retained the deprecated bit :)

balazsgaspar and others added 30 commits March 7, 2021 18:55

Update README.adoc

22bb476

Fixed a typo ("yum" -> "git")

Merge pull request #1 from cloudera-labs/trunk

0c509cd

Merge upstream changes

Added images for streaming lite labs

7964019

Added images for Lab 6 instructions

76d4839

Added images for Lab 6 instructions

62ceb44

Added automated Viz setup - alpha

0794330

Merging streaming_flink and streaming_lite content from zBrainiac/edg…

b95615d

…e2ai-workshop

Merging addional images for streaming lite workshop from zBrainiac/ed…

bce14f4

…ge2ai-workshop

Enabled unauthenticated app access. Added viz app status / health check.

a0014b1

Update streaming_lite.adoc

6b40d43

Change Lab 6 from Hue to CDP Data Visualization

Added images for Lab 6 instructions

005ece3

Updated web forms to improve registration process, fixed small issue …

923244f

…in cdsw setup script.

Update streaming_lite.adoc

e77f12b

Updated Lab 6 instructions for CDV

Update streaming_lite.adoc

c734153

Update streaming_lite.adoc

aa17b26

Update streaming_lite.adoc

38ffc94

Update streaming_lite.adoc

4ce7267

Fixing Kudu timestamp column issue

Update streaming_lite.adoc

ce77a76

Added images for Lab 6 instructions

dd43cfe

Added images for Lab 6 instructions

3dad2c2

Update streaming_lite.adoc

8699895

Update streaming_lite.adoc

4cd36e8

Added images for Lab 6 instructions

833e52e

Added images for Lab 6 instructions

a5a14a0

Update streaming_lite.adoc

2f4fb89

Added images for Lab 6 instructions

b8db4e3

Added images for Lab 6 instructions

823773c

Update streaming_lite.adoc

d215fc8

Added images for Lab 6 instructions

04143dd

Update streaming_lite.adoc

085827f

Balazs Gaspar and others added 6 commits April 19, 2021 18:13

Typo fix

d5db2b6

Update streaming_lite.adoc

09c956a

Updated images for Lab 7 instructions

fe0b3e4

Updated Lab 7 instructions for lite workshop

b13b720

Modified source topic name from iot_enriched to iot (for lite labs)

a67f988

Update streaming_lite.adoc

b5c8e65

asdaraujo requested changes May 21, 2021

View reviewed changes

asdaraujo reviewed May 21, 2021

View reviewed changes

asdaraujo requested changes May 21, 2021

View reviewed changes

asdaraujo reviewed May 24, 2021

View reviewed changes

Balazs Gaspar added 9 commits July 12, 2021 16:23

Resolving conflicts with upstream

00859ae

Resolving conflicts with upstream

e8f5a7b

Resolving conflicts with upstream

c4e9fe2

Resolving conflicts with upstream

eb4b808

Resolving conflicts with upstream

57fff52

Resolving conflicts with upstream

8bfbfdb

Resolving conflicts with upstream

ddbba15

Resolving conflicts with upstream

e67971a

Resolving conflicts with upstream

0c555f7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge balazs-patchset #13

Merge balazs-patchset #13

balazsgaspar commented Apr 19, 2021

asdaraujo commented May 21, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

balazsgaspar Jul 12, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

balazsgaspar Jul 12, 2021

asdaraujo May 21, 2021

balazsgaspar Jul 12, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

balazsgaspar Jul 12, 2021

asdaraujo left a comment

asdaraujo May 21, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021

asdaraujo May 21, 2021 •

edited

Loading

asdaraujo May 21, 2021

asdaraujo May 24, 2021

Merge balazs-patchset #13

Are you sure you want to change the base?

Merge balazs-patchset #13

Conversation

balazsgaspar commented Apr 19, 2021

asdaraujo commented May 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asdaraujo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asdaraujo May 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asdaraujo May 21, 2021 •

edited

Loading