Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rc 0.0.1 #27

Merged
merged 41 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
33e7a71
Added pre-commit and ruff
spikelynch Nov 20, 2024
fe6bab4
Ruff reformatting
spikelynch Nov 20, 2024
018b99f
pre-commit config
spikelynch Nov 20, 2024
3d72f91
Very basic test
spikelynch Nov 26, 2024
8e41a9b
Testing pre-commit
spikelynch Nov 26, 2024
9fb18e7
Added ruff linting workflow
spikelynch Nov 26, 2024
ab0803f
Merge pull request #9 from Sydney-Informatics-Hub/feature-static-checks
spikelynch Nov 26, 2024
6d7ae8f
Added pytest CI
spikelynch Nov 26, 2024
633779a
Correct test name for pytest
spikelynch Nov 26, 2024
eab55d1
Merge pull request #10 from Sydney-Informatics-Hub/feature-test-ci
spikelynch Nov 26, 2024
ce6c7ab
Simple script to make test crates
spikelynch Dec 2, 2024
792cd09
Test crates
spikelynch Dec 2, 2024
e339e8e
moved test minimal crate
spikelynch Dec 2, 2024
f395b5a
✨REFACTORING✨ - it's now a class, and I've standardised things like
spikelynch Dec 3, 2024
10d0bdc
Merge pull request #13 from Sydney-Informatics-Hub/bugfix-big-crate
spikelynch Dec 3, 2024
57c6910
First attempt at fuzz testing (not actually checking its outputs)
spikelynch Dec 3, 2024
fafcf61
Ok it's at the stage where it's failing in interesting ways
spikelynch Dec 3, 2024
895b2df
Fixed typo in main script
spikelynch Dec 3, 2024
0e902b1
Added datePublished to minimal crate
spikelynch Dec 3, 2024
7e4052d
Including external ids as target_ids - this came up to make the
spikelynch Dec 3, 2024
cf85e36
Corrected error in workflow for pytest
spikelynch Dec 3, 2024
09f0090
Merge pull request #14 from Sydney-Informatics-Hub/feature-real-tests
spikelynch Dec 3, 2024
4c8ffb0
Moved the JSONCrate into src as TinyCrate
spikelynch Dec 4, 2024
b3b76a9
removed rocrate.py, now using tinycrate
spikelynch Dec 4, 2024
1b0ada0
Merge pull request #22 from Sydney-Informatics-Hub/feature-use-new-ro…
spikelynch Dec 4, 2024
95763c4
Made it into a proper pyproject.toml script
spikelynch Dec 4, 2024
367c5f4
Forgot to rename the modules
spikelynch Dec 5, 2024
70385d4
Merge pull request #25 from Sydney-Informatics-Hub/feature-use-new-ro…
spikelynch Dec 5, 2024
a9c9f2a
Added a TinyEntity class around the entities in the graph
spikelynch Dec 5, 2024
bdc07b2
Broke out the randomised text functions as fuzz.py, added minimal
spikelynch Dec 5, 2024
46a9bb2
Fixed typo in method name
spikelynch Dec 5, 2024
9af8e57
Very basic round-trip tests which pass
spikelynch Dec 5, 2024
50d12ba
File and url loader
spikelynch Dec 5, 2024
36bbc84
Passing http tests
spikelynch Dec 5, 2024
9fe64f9
Added text file for tests
spikelynch Dec 5, 2024
1896b2d
Updated the test crate with an indexableText property to textfile.txt
spikelynch Dec 5, 2024
a1d671d
Starting to make fetch happen
spikelynch Dec 5, 2024
037af21
Tabulator can fetch text from a file on disk
spikelynch Dec 5, 2024
2191e4d
Loads crates and --text configured properties from URLs
spikelynch Dec 5, 2024
dd743a9
Fixed pathlib bug
spikelynch Dec 5, 2024
c1468ad
Merge pull request #26 from Sydney-Informatics-Hub/feature-abstract-c…
spikelynch Dec 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: linters

on:
push:
branches:
- main
pull_request:
branches:
- '**'

jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Check out source
uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "0.5.4"

- name: Set up Python
run: uv python install

- name: Install project
run: uv sync --all-extras --dev

- name: ruff
run: uv run ruff check --output-format=github
30 changes: 30 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: pytest

on:
push:
branches:
- main
pull_request:
branches:
- '**'

jobs:
pytest:
runs-on: ubuntu-latest
steps:
- name: Check out source
uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "0.5.4"

- name: Set up Python
run: uv python install

- name: Install project
run: uv sync --all-extras --dev

- name: Pytest
run: uv run pytest
9 changes: 9 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.7.4
hooks:
# Run the linter.
- id: ruff
# Run the formatter.
- id: ruff-format
15 changes: 1 addition & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,6 @@ Install [uv](https://docs.astral.sh/uv/), then

## Usage

To build an SQLite version of a crate:
> uv run src/rocrate_tabular/rocrate_tabular.py path/to/crate crate.db

> uv run src/rocrate_tabular/rocrate_tabular.py -c path/to/crate -o out.db

To build a CSV version of the same tables:

> uv run src/rocrate_tabular/rocrate_tabular.py -c path/to/crate -o out.csv


## Todo

Build this out so it's a better library:

- [ ] unit tests
- [ ] better separation of output formats
- [ ] build tables for kinds of crate entities with sensible defaults
17 changes: 16 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,25 @@ description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"rocrate>=0.11.0",
"requests>=2.32.3",
"sqlite-utils>=3.37",
]

[project.scripts]
tabulator = "rocrate_tabular.tabulator:cli"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv]
dev-dependencies = [
"pre-commit>=4.0.1",
"pytest>=8.3.3",
"ruff>=0.7.4",
]

[dependency-groups]
dev = [
"pytest-httpserver>=1.1.0",
]
Loading
Loading