wasm + shinylive #243
Merged
wasm + shinylive #243
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CLA Assistant Lite bot ✅ All contributors have signed the CLA |
|
I have read the CLA Document and I hereby sign the CLA |
|
This is so cool! Let me test it out locally and also investigate the deployment issues you're referring to. |
Melkiades
reviewed
Apr 16, 2024
Honestly - I don't know. By definition, this is executed in the browser so it's very much depends on the user local machine. I'm afraid there is little we can do about this... I will try to research this more |
|
Hi All, I have pushed an alternative way of embedding webR - inside collapsible panel instead of a tab. Please let me know which one is better
webr-table-callout.mp4webr-app-callout.mp4These two exists in a separate articles so you can render and try this out yourself. (I would actually recommend to do it) Please share your feedback with emojis: |
|
I'm having a hard time running this locally. The error log is so long that I can't see the beginning, which usually gives me clues about any missed dependencies. I'm pretty sure I'm just missing some dependencies. So far, I've installed shinylive, webr, and webshot. Incomplete Error Loglocal function resolveDependencyLinkTags(linkTags)
if linkTags ~= nil then
for i, v in ipairs(linkTags) do
v.href = resolvePath(v.href)
end
return linkTags
else
return nil
end
end
-- Convert dependency files which may be just a string (path) or
-- incomplete objects into valid file dependencies
local function resolveFileDependencies(name, dependencyFiles)
if dependencyFiles ~= nil then
-- make sure this is an array
if type(dependencyFiles) ~= "table" or not utils.table.isarray(dependencyFiles) then
error("Invalid HTML Dependency: " .. name .. " property must be an array")
end
local finalDependencies = {}
for i, v in ipairs(dependencyFiles) do
if type(v) == "table" then
-- fill in the name, if one is not provided
if v.name == nil then
v.name = pandoc.path.filename(v.path)
end
finalDependencies[i] = v
elseif type(v) == "string" then
-- turn a string into a name and path
finalDependencies[i] = {
name = pandoc.path.filename(v),
path = v
}
else
-- who knows what this is!
error("Invalid HTML Dependency: " .. name .. " property contains an unexpected type.")
end
end
return finalDependencies
else
return nil
end
end
-- Convert dependency files which may be just a string (path) or
-- incomplete objects into valid file dependencies
local function resolveServiceWorkers(serviceworkers)
if serviceworkers ~= nil then
-- make sure this is an array
if type(serviceworkers) ~= "table" or not utils.table.isarray(serviceworkers) then
error("Invalid HTML Dependency: serviceworkers property must be an array")
end
local finalServiceWorkers = {}
for i, v in ipairs(serviceworkers) do
if type(v) == "table" then
-- fill in the destination as the root, if one is not provided
if v.source == nil then
error("Invalid HTML Dependency: a serviceworker must have a source.")
else
v.source = resolvePathExt(v.source)
end
finalServiceWorkers[i] = v
elseif type(v) == "string" then
-- turn a string into a name and path
finalServiceWorkers[i] = {
source = resolvePathExt(v)
}
else
-- who knows what this is!
error("Invalid HTML Dependency: serviceworkers property contains an unexpected type.")
end
end
return finalServiceWorkers
else
return nil
end
end
local latexTableWithOptionsPattern = "(\\begin{table}%[[^%]]+%])(.*)(\\end{table})"
local latexTablePattern = "(\\begin{table})(.*)(\\end{table})"
local latexLongtablePatternwWithPosAndAlign = "(\\begin{longtable}%[[^%]]+%]{[^\n]*})(.*)(\\end{longtable})"
local latexLongtablePatternWithPos = "(\\begin{longtable}%[[^%]]+%])(.*)(\\end{longtable})"
local latexLongtablePatternWithAlign = "(\\begin{longtable}{[^\n]*})(.*)(\\end{longtable})"
local latexLongtablePattern = "(\\begin{longtable})(.*)(\\end{longtable})"
local latexTabularPatternWithPosAndAlign = "(\\begin{tabular}%[[^%]]+%]{[^\n]*})(.*)(\\end{tabular})"
local latexTabularPatternWithPos = "(\\begin{tabular}%[[^%]]+%])(.*)(\\end{tabular})"
local latexTabularPatternWithAlign = "(\\begin{tabular}{[^\n]*})(.*)(\\end{tabular})"
local latexTabularPattern = "(\\begin{tabular})(.*)(\\end{tabular})"
local latexCaptionPattern = "(\\caption{)(.-)(}[^\n]*\n)"
local latexTablePatterns = pandoc.List({
latexTableWithOptionsPattern,
latexTablePattern,
latexLongtablePatternwWithPosAndAlign,
latexLongtablePatternWithPos,
latexLongtablePatternWithAlign,
latexLongtablePattern,
latexTabularPatternWithPosAndAlign,
latexTabularPatternWithPos,
latexTabularPatternWithAlign,
latexTabularPattern,
})
-- global quarto params
local paramsJson = base64.decode(os.getenv("QUARTO_FILTER_PARAMS"))
local quartoParams = json.decode(paramsJson)
function param(name, default)
local value = quartoParams[name]
if value == nil then
value = default
end
return value
end
local function projectDirectory()
return os.getenv("QUARTO_PROJECT_DIR")
end
local function projectOutputDirectory()
local outputDir = param("project-output-dir", "")
local projectDir = projectDirectory()
if projectDir then
return pandoc.path.join({projectDir, outputDir})
else
return nil
end
end
-- Provides the project relative path to the current input
-- if this render is in the context of a project
local function projectRelativeOutputFile()
-- the project directory
local projDir = projectDirectory()
-- the offset to the project
if projDir then
-- relative from project directory to working directory
local workingDir = pandoc.system.get_working_directory()
local projRelFolder = pandoc.path.make_relative(workingDir, projDir, false)
-- add the file output name and normalize
local projRelPath = pandoc.path.join({projRelFolder, PANDOC_STATE['output_file']})
return pandoc.path.normalize(projRelPath);
else
return nil
end
end
local function inputFile()
local source = param("quarto-source", "")
if pandoc.path.is_absolute(source) then
return source
else
local projectDir = projectDirectory()
if projectDir then
return pandoc.path.join({projectDir, source})
else
-- outside of a project, quarto already changes
-- pwd to the file's directory prior to calling pandoc,
-- so we should just use the filename
-- https://github.com/quarto-dev/quarto-cli/issues/7424
local path_parts = pandoc.path.split(source)
return pandoc.path.join({pandoc.system.get_working_directory(), path_parts[#path_parts]})
end
end
end
local function outputFile()
local projectOutDir = projectOutputDirectory()
if projectOutDir then
local projectDir = projectDirectory()
if projectDir then
local input = pandoc.path.directory(inputFile())
local relativeDir = pandoc.path.make_relative(input, projectDir)
if relativeDir and relativeDir ~= '.' then
return pandoc.path.join({projectOutDir, relativeDir, PANDOC_STATE['output_file']})
end
end
return pandoc.path.join({projectOutDir, PANDOC_STATE['output_file']})
else
return pandoc.path.join({pandoc.system.get_working_directory(), PANDOC_STATE['output_file']})
end
end
local function version()
local versionString = param('quarto-version', 'unknown')
local success, versionObject = pcall(pandoc.types.Version, versionString)
if success then
return versionObject
else
return versionString
end
end
local function projectProfiles()
return param('quarto_profile', {})
end
local function projectOffset()
return param('project-offset', nil)
end
local function file_exists(name)
local f = io.open(name, 'r')
if f ~= nil then
io.close(f)
return true
else
return false
end
end
local function write_file(path, contents, mode)
pandoc.system.make_directory(pandoc.path.directory(path), true)
mode = mode or "a"
local file = io.open(path, mode)
if file then
file:write(contents)
file:close()
return true
else
return false
end
end
local function read_file(path)
local file = io.open(path, "rb")
if not file then return nil end
local content = file:read "*a"
file:close()
return content
end
local function remove_file(path)
return os.remove(path)
end
-- Quarto internal module - makes functions available
-- through the filters
_quarto = {
processDependencies = processDependencies,
format = format,
patterns = {
latexTabularPattern = latexTabularPattern,
latexTablePattern = latexTablePattern,
latexLongtablePattern = latexLongtablePattern,
latexTablePatterns = latexTablePatterns,
latexCaptionPattern = latexCaptionPattern
},
utils = utils,
withScriptFile = function(file, callback)
table.insert(scriptFile, file)
local result = callback()
table.remove(scriptFile, #scriptFile)
return result
end,
projectOffset = projectOffset,
file = {
read = read_file,
write = function(path, contents)
return write_file(path, contents, "wb")
end,
write_text = function(path, contents)
return write_file(path, contents, "a")
end,
exists = file_exists,
remove = remove_file
}
}
-- this injection here is ugly but gets around
-- a hairy order-of-import issue that would otherwise happen
-- because string_to_inlines requires some filter code that is only
-- later imported
_quarto.utils.string_to_inlines = function(s)
return string_to_quarto_ast_inlines(s)
end
_quarto.utils.string_to_blocks = function(s)
return string_to_quarto_ast_blocks(s)
end
_quarto.utils.render = function(n)
return _quarto.ast.walk(n, render_extended_nodes())
end
-- The main exports of the quarto module
quarto = {
doc = {
add_html_dependency = function(htmlDependency)
-- validate the dependency
if htmlDependency.name == nil then
error("HTML dependencies must include a name")
end
if htmlDependency.meta == nil and
htmlDependency.links == nil and
htmlDependency.scripts == nil and
htmlDependency.stylesheets == nil and
htmlDependency.resources == nil and
htmlDependency.serviceworkers == nil and
htmlDependency.head == nil then
error("HTML dependencies must include at least one of meta, links, scripts, stylesheets, serviceworkers, or resources. All appear empty.")
end
-- validate that the meta is as expected
if htmlDependency.meta ~= nil then
if type(htmlDependency.meta) ~= 'table' then
error("Invalid HTML Dependency: meta value must be a table")
elseif utils.table.isarray(htmlDependency.meta) then
error("Invalid HTML Dependency: meta value must must not be an array")
end
end
-- validate link tags
if htmlDependency.links ~= nil then
if type(htmlDependency.links) ~= 'table' or not utils.table.isarray(htmlDependency.links) then
error("Invalid HTML Dependency: links must be an array")
else
for i, v in ipairs(htmlDependency.links) do
if type(v) ~= "table" or (v.href == nil or v.rel == nil) then
error("Invalid HTML Dependency: each link must be a table containing both rel and href properties.")
end
end
end
end
-- resolve names so they aren't required
htmlDependency.scripts = resolveFileDependencies("scripts", htmlDependency.scripts)
htmlDependency.stylesheets = resolveFileDependencies("stylesheets", htmlDependency.stylesheets)
htmlDependency.resources = resolveFileDependencies("resources", htmlDependency.resources)
-- pass the dependency through to the file
writeToDependencyFile(dependency("html", {
name = htmlDependency.name,
version = htmlDependency.version,
external = true,
meta = htmlDependency.meta,
links = resolveDependencyLinkTags(htmlDependency.links),
scripts = resolveDependencyFilePaths(htmlDependency.scripts),
stylesheets = resolveDependencyFilePaths(htmlDependency.stylesheets),
resources = resolveDependencyFilePaths(htmlDependency.resources),
serviceworkers = resolveServiceWorkers(htmlDependency.serviceworkers),
head = htmlDependency.head,
}))
end,
attach_to_dependency = function(name, pathOrFileObj)
if name == nil then
fail("The target dependency name for an attachment cannot be nil. Please provide a valid dependency name.")
end
-- path can be a string or an obj { name, path }
local resolvedFile = {}
if type(pathOrFileObj) == "table" then
-- validate that there is at least a path
if pathOrFileObj.path == nil then
fail("Error attaching to dependency '" .. name .. "'.\nYou must provide a 'path' when adding an attachment to a dependency.")
end
-- resolve a name, if one isn't provided
local name = pathOrFileObj.name
if name == nil then
name = pandoc.path.filename(pathOrFileObj.path)
end
-- the full resolved file
resolvedFile = {
name = name,
path = resolvePathExt(pathOrFileObj.path)
}
else
resolvedFile = {
name = pandoc.path.filename(pathOrFileObj),
path = resolvePathExt(pathOrFileObj)
}
end
writeToDependencyFile(dependency("html-attachment", {
name = name,
file = resolvedFile
}))
end,
use_latex_package = function(package, options)
writeToDependencyFile(dependency("usepackage", {package = package, options = options }))
end,
add_format_resource = function(path)
writeToDependencyFile(dependency("format-resources", { file = resolvePathExt(path)}))
end,
add_resource = function(path)
writeToDependencyFile(dependency("resources", { file = resolvePathExt(path)}))
end,
add_supporting = function(path)
writeToDependencyFile(dependency("supporting", { file = resolvePathExt(path)}))
end,
include_text = function(location, text)
writeToDependencyFile(dependency("text", { text = text, location = resolveLocation(location)}))
end,
include_file = function(location, path)
writeToDependencyFile(dependency("file", { path = resolvePathExt(path), location = resolveLocation(location)}))
end,
is_format = format.isFormat,
cite_method = function()
local citeMethod = param('cite-method', 'citeproc')
return citeMethod
end,
pdf_engine = function()
local engine = param('pdf-engine', 'pdflatex')
return engine
end,
has_bootstrap = function()
local hasBootstrap = param('has-bootstrap', false)
return hasBootstrap
end,
is_filter_active = function(filter)
return quarto_global_state.active_filters[filter]
end,
output_file = outputFile(),
input_file = inputFile(),
crossref = {}
},
project = {
directory = projectDirectory(),
offset = projectOffset(),
profile = pandoc.List(projectProfiles()),
output_directory = projectOutputDirectory()
},
utils = {
dump = utils.dump,
table = utils.table,
type = utils.type,
resolve_path = resolvePathExt,
resolve_path_relative_to_document = resolvePath,
as_inlines = utils.as_inlines,
as_blocks = utils.as_blocks,
string_to_blocks = utils.string_to_blocks,
string_to_inlines = utils.string_to_inlines,
render = utils.render,
match = utils.match,
add_to_blocks = utils.add_to_blocks
},
json = json,
base64 = base64,
log = logging,
version = version()
}
-- alias old names for backwards compatibility
quarto.doc.addHtmlDependency = quarto.doc.add_html_dependency
quarto.doc.attachToDependency = quarto.doc.attach_to_dependency
quarto.doc.useLatexPackage = quarto.doc.use_latex_package
quarto.doc.addFormatResource = quarto.doc.add_format_resource
quarto.doc.includeText = quarto.doc.include_text
quarto.doc.includeFile = quarto.doc.include_file
quarto.doc.isFormat = quarto.doc.is_format
quarto.doc.citeMethod = quarto.doc.cite_method
quarto.doc.pdfEngine = quarto.doc.pdf_engine
quarto.doc.hasBootstrap = quarto.doc.has_bootstrap
quarto.doc.project_output_file = projectRelativeOutputFile
quarto.utils.resolvePath = quarto.utils.resolve_path
-- since Pandoc 3, pandoc.Null is no longer an allowed constructor.
-- this workaround makes it so that our users extensions which use pandoc.Null
-- still work, assuming they call pandoc.Null() in a "simple" way.
pandoc.Null = function()
return {}
end
:1537) An error occurred:
Error reading dependencies from /Users/unardid/Documents/dsx/insightengineering/tlg-catalog/book/_extensions/coatless/webr/qwebr-monaco-editor-init.html
Error running filter /Applications/quarto/share/filters/main.lua:
/Applications/quarto/share/filters/main.lua:2246: attempt to call a nil value (global 'crash_with_stack_trace')
stack traceback:
/Applications/quarto/share/filters/main.lua:1826: in function 'fail'
[string "..."]:1537: in upvalue 'processFileDependency'
[string "..."]:1602: in field 'processDependencies'
/Applications/quarto/share/filters/main.lua:7356: in field 'Meta'
/Applications/quarto/share/filters/main.lua:240: in function 'run_emulated_filter'
/Applications/quarto/share/filters/main.lua:942: in local 'callback'
/Applications/quarto/share/filters/main.lua:960: in upvalue 'run_emulated_filter_chain'
/Applications/quarto/share/filters/main.lua:996: in function </Applications/quarto/share/filters/main.lua:993> |
|
Thanks Dony for checking this. After your message I noticed that one file is missing because it was git-ignored. I added this to the repo and this should work. Can you please try again now? |
|
Thanks @pawelru. It works for me now. PerformanceI noticed that the teal app is performing quite slow, for example when adding a filter, during the shinylive session. It felt like there's a delay to the reactivity. DesignWhen I see a panel, I typically perceive it as a notification, and it's not clear that I have to click the panel or the arrow button on the far right, which then reveals another major interaction. Thanks for exploring this! This is interesting to see and we should look into this further. |
|
Thank you Dony for a good feedback. I can confirm that I am also observing some delays. This is probably coming from webR itself. I cannot spot any settings that could impact performance directly so if this is (will be) a big problem then we would have to dig much deeper.
Note that we can work on panel CSS styling and / or the displayed text label (e.g. "Click here to try this out in WebR"). Would this change your view on that? Let's also see what others will say. |
|
first of all, this is an absolutely amazing feature! i prefer the tab version, as the collapse thing might just get too long and confusing, |
Yeah, I'm always open to seeing what it looks like if we had a chance to design this. If it makes sense for everyone, I'm fine with going with it. I don't have a strong preference here. By the way, I did try to get the feel if the editor and the teal app are side-by-side in full screen. The full screen can be achieved by using the I like this format better but I wish there's a way to set the editor's width when loaded. I want to make it smaller so we can see the teal's app in full view. |
From what I've observed, Once webR and shinylive finished with loading the packages, the code will run automatically and generate the static tables and teal app. But if we're using the static output for unit testing, then yes, maybe we should keep them. It sucks because the static output for teal app is forcing the tabset panel to be two levels. If we don't have static output for teal app, we can expand the view to not use the right margin like so: This view already looks good to me and will look even better on a wide-screen monitor. |
There is one more important thing against going webr only which is when the error occurs. Under webr-only scenario, all the code will be evaluated only in the user browser and the book rendering process will be always successful. We have to have this code evaluated during render as well. I actually have one more idea - eval and don't include. Let me try this one as well |
|
OK I have pushed a new version with a separate article using webr-only approach. Please have a look. I personally don't really like that. It's slow (one needs to wait for packages install and code eval to see the results), not so nice as quarto-rendered outputs and it's quite complicated (it's not super obvious to add I was also experimenting with the layout width for shinylive apps with no success. If I extend the width of this callout/tab then it makes the width of the overall table a little bit smaller. That's some bad side-effect that I don't know how to control I have abandoned this for now |
|
It seems that there is a preference towards the tabs so I'm going to continue with that for all the rest of articles |
I see your point on this and I agree. Because webR takes a while to load, we shouldn't have to force users to wait for the webR/shinylive to run if they don't want to use it and just want to see the result quickly. In the end, I like what you did with |
|
This is ready to be reviewed 🚀 |
pawelru
commented
Apr 29, 2024
Unit Tests Summary341 tests 0 ✅ 56s ⏱️ Results for commit 0d254be. ♻️ This comment has been updated with latest results. |
Unit Test Performance DifferenceTest suite performance difference
Additional test case details
Results for commit f25ae25 ♻️ This comment has been updated with latest results. |
|
I forked the repo in order to test deployment before the merge. I discovered some issues when installing |
cicdguy
approved these changes
May 6, 2024
This reverts commit 939529e.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
close #32
tlg-catalog-table.mp4
tlg-catalog-app.mp4
This is done for one article only but it's assumed that once green-lighted this will be copy&pasted for other articles as well.
Please share your thoughts on:
I have experimented a lot and have found out that the tab-based layout looks the best for me. I considered vertical alignment (static output above webR or vice versa) but then if one executes webR there will be two identical outputs one after another which looks odd.
I have also considered having wasm alone (i.e. without static outputs) but this would force users to click "run" to see the outcome as well as eliminate testability of the article code (i.e. run-on-demand vs run-on-render). Therefore I think we have to have the "static" part.
@cicdguy I might need some help to test whether deployed version is fully functional as well as the deployment process itself. In particular I have found out that folks observed some issues like this. I would appreciate your help on this. Feel free to push commits if needed.
TODO: