Skip to content

Commit

Permalink
Merge branch 'master' into fix-sticky-issue
Browse files Browse the repository at this point in the history
  • Loading branch information
aglinxinyuan authored Dec 17, 2024
2 parents 44d531f + 41c47bc commit e9eeb8a
Show file tree
Hide file tree
Showing 13 changed files with 103 additions and 430 deletions.
64 changes: 2 additions & 62 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
<br>

<h4 align="center">
<a href="https://github.com/Texera/texera#videos">Demo Video</a>
<a href="https://texera.io">Official Site</a>
|
<a href="https://texera.github.io/blog/">Blogs</a>
<a href="https://texera.io/?cat=11">Blogs</a>
|
<a href="https://github.com/Texera/texera/wiki/Getting-Started">Getting Started</a>
<br>
Expand All @@ -29,13 +29,6 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Largest_Deployment-100_nodes,_400_cores-green">
</p>

# Motivation

* Data science is labor-intensive and particularly challenging for non-IT users applying AI/ML.
* Many workflow-based data science platforms lack parallelism, limiting their ability to handle big datasets.
* Cloud services and technologies have advanced significantly over the past decade, enabling powerful browser-based interfaces supported by high-speed networks.
* Existing data science platforms offer limited interaction during long-running jobs, making them difficult to manage after execution begins.

# Goals

* Provide data science as cloud services;
Expand Down Expand Up @@ -148,59 +141,6 @@ The workflow in the use case shown below includes data cleaning, ML model traini
_In JAMIA 2021_ | [PDF](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7989302/pdf/ocab047.pdf)
</details>


# Education
<table>
<tr style="height: 500px;">
<td align="center">
<a href="https://ds4all.ics.uci.edu/">
<img src="https://ds4all.ics.uci.edu/wp-content/uploads/2023/07/banner-1024x576.png">
</a>
<p><b>Data Science for All</b></p>
An NSF-funded summer program to teach high-school students data science and AI/ML
</td>
<td align="center">
<a href="https://canvas.eee.uci.edu/courses/63639/pages/syllabus">
<img src="https://github.com/user-attachments/assets/a7569fd3-6857-48b4-80dc-d9f006ae2c8f">
</a>
<p><b>ICS 80: Data Science and AI/ML Using Workflows</b></p>
A Spring 2024 course at UCI, teaching 42 undergraduates, most of whom are not computer science majors, to learn data science and AI/ML
</td>
<td align="center">
<a href="https://sites.google.com/uci.edu/ds-workshop2024/home">
<img src="https://www.cerritos.edu/_resources/images/common/cerritos-college-logo.svg">
</a>
<p><b>Workshop of Data Science for Everyone at Cerritos College</b></p>
A two-day workshop designed for non-CS students to learn data science and ML without a single line of coding
</td>
</tr>
</table>


# Videos
<table>
<tr style="height: 500px;">
<td align="center">
<a href="https://www.youtube.com/watch?v=B81iMFS5fPc">
<img src="https://img.youtube.com/vi/B81iMFS5fPc/0.jpg" alt="Watch the video">
</a>
<p><b>dkNET Webinar 04/26/2024</b></p>
</td>
<td align="center">
<a href="https://www.youtube.com/watch?v=SP-XiDADbw0">
<img src="https://img.youtube.com/vi/SP-XiDADbw0/0.jpg" alt="Watch the video">
</a>
<p><b>Texera Demo @ VLDB'20</b></p>
</td>
<td align="center">
<a href="https://www.youtube.com/watch?v=T5ShFRfHmgI">
<img src="https://img.youtube.com/vi/T5ShFRfHmgI/0.jpg" alt="Watch the video">
</a>
<p><b>Amber Presentation @ VLDB'20</b></p>
</td>
</tr>
</table>

# Getting Started

* For users, visit [Guide to Use Texera](https://github.com/Texera/texera/wiki/Getting-Started).
Expand Down
1 change: 1 addition & 0 deletions core/dao/build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -88,4 +88,5 @@ libraryDependencies ++= Seq(

libraryDependencies ++= Seq(
"mysql" % "mysql-connector-java" % "8.0.33", // MySQL connector
"org.yaml" % "snakeyaml" % "1.30", // for reading storage config yaml file
)
46 changes: 46 additions & 0 deletions core/dao/src/main/resources/jooq-conf.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<configuration>
<generator>
<!-- The generator for dao -->
<generate>
<generatedAnnotation>false</generatedAnnotation>

<daos>true</daos>
<interfaces>true</interfaces>
</generate>
<!-- The default code generator. You can override this one, to generate your own code style.
Supported generators:
- org.jooq.codegen.JavaGenerator
- org.jooq.codegen.ScalaGenerator
Defaults to org.jooq.codegen.JavaGenerator -->
<name>org.jooq.codegen.JavaGenerator</name>

<database>
<!-- The database type. The format here is:
org.jooq.meta.[database].[database]Database -->
<name>org.jooq.meta.mysql.MySQLDatabase</name>

<!-- The database schema (or in the absence of schema support, in your RDBMS this
can be the owner, user, database name) to be generated -->
<inputSchema>texera_db</inputSchema>

<!-- All elements that are generated from your schema
(A Java regular expression. Use the pipe to separate several expressions)
Watch out for case-sensitivity. Depending on your database, this might be important! -->
<includes>.*</includes>

<!-- All elements that are excluded from your schema
(A Java regular expression. Use the pipe to separate several expressions).
Excludes match before includes, i.e. excludes have a higher priority -->
<excludes>(test_.*)|(ignore_.*)</excludes>

</database>

<target>
<!-- The destination package of your generated classes (within the destination directory) -->
<packageName>edu.uci.ics.texera.dao.jooq.generated</packageName>

<!-- The destination directory of your generated classes. Using Maven directory layout here -->
<directory>dao/src/main/scala</directory>
</target>
</generator>
</configuration>
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
package edu.uci.ics.texera.dao

import org.jooq.codegen.GenerationTool
import org.jooq.meta.jaxb.{Configuration, Jdbc}
import org.yaml.snakeyaml.Yaml

import java.io.InputStream
import java.nio.file.{Files, Path}
import java.util.{Map => JMap}
import scala.jdk.CollectionConverters._

object JooqCodeGenerator {
@throws[Exception]
def main(args: Array[String]): Unit = {
// Load jOOQ configuration XML
val jooqXmlPath: Path =
Path.of("dao").resolve("src").resolve("main").resolve("resources").resolve("jooq-conf.xml")
val jooqConfig: Configuration = GenerationTool.load(Files.newInputStream(jooqXmlPath))

// Load YAML configuration
val yamlConfPath: Path = Path
.of("workflow-core")
.resolve("src")
.resolve("main")
.resolve("resources")
.resolve("storage-config.yaml")
val yaml = new Yaml
val inputStream: InputStream = Files.newInputStream(yamlConfPath)

val conf: Map[String, Any] =
yaml.load(inputStream).asInstanceOf[JMap[String, Any]].asScala.toMap

val jdbcConfig = conf("storage")
.asInstanceOf[JMap[String, Any]]
.asScala("jdbc")
.asInstanceOf[JMap[String, Any]]
.asScala

// Set JDBC configuration for jOOQ
val jooqJdbcConfig = new Jdbc
jooqJdbcConfig.setDriver("com.mysql.cj.jdbc.Driver")
jooqJdbcConfig.setUrl(jdbcConfig("url").toString)
jooqJdbcConfig.setUsername(jdbcConfig("username").toString)
jooqJdbcConfig.setPassword(jdbcConfig("password").toString)

jooqConfig.setJdbc(jooqJdbcConfig)

// Generate the code
GenerationTool.generate(jooqConfig)

// Close input stream
inputStream.close()
}
}
Binary file not shown.
18 changes: 0 additions & 18 deletions core/util/build.sbt

This file was deleted.

89 changes: 0 additions & 89 deletions core/util/conf/jooq-conf.xml

This file was deleted.

1 change: 0 additions & 1 deletion core/util/project/build.properties

This file was deleted.

47 changes: 0 additions & 47 deletions core/util/src/main/java/edu/uci/ics/util/RunCodegen.java

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ import edu.uci.ics.amber.operator.cartesianProduct.CartesianProductOpDesc
import edu.uci.ics.amber.operator.dictionary.DictionaryMatcherOpDesc
import edu.uci.ics.amber.operator.difference.DifferenceOpDesc
import edu.uci.ics.amber.operator.distinct.DistinctOpDesc
import edu.uci.ics.amber.operator.download.BulkDownloaderOpDesc
import edu.uci.ics.amber.operator.dummy.DummyOpDesc
import edu.uci.ics.amber.operator.filter.SpecializedFilterOpDesc
import edu.uci.ics.amber.operator.hashJoin.HashJoinOpDesc
Expand Down Expand Up @@ -202,7 +201,6 @@ trait StateTransferFunc
new Type(value = classOf[RedditSearchSourceOpDesc], name = "RedditSearch"),
new Type(value = classOf[PythonLambdaFunctionOpDesc], name = "PythonLambdaFunction"),
new Type(value = classOf[PythonTableReducerOpDesc], name = "PythonTableReducer"),
new Type(value = classOf[BulkDownloaderOpDesc], name = "BulkDownloader"),
new Type(value = classOf[URLFetcherOpDesc], name = "URLFetcher"),
new Type(value = classOf[CartesianProductOpDesc], name = "CartesianProduct"),
new Type(value = classOf[FilledAreaPlotOpDesc], name = "FilledAreaPlot"),
Expand Down
Loading

0 comments on commit e9eeb8a

Please sign in to comment.