Skip to content

Commit

Permalink
Add release log to GitHub (#170)
Browse files Browse the repository at this point in the history
  • Loading branch information
@erikcs
erikcs authored Feb 16, 2024
1 parent 4eef4df commit 3b8ca17
Showing 1 changed file with 377 additions and 1 deletion.
378 changes: 377 additions & 1 deletion releases/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
### Preparing a new CRAN release
# policytree Release Process

This is just a minor summary document, practically all details are the same as the [GRF release process](https://github.com/grf-labs/grf/tree/master/releases). Memory/perf timings for releases are recorded at the end.

0. Open a "Prepare the v.x release PR" and update CHANGELOG

Expand Down Expand Up @@ -39,3 +41,377 @@
git tag -a v1.1.0 "release-commit-hash"
git push --tags
```

Make sure to submit a test tarball containing everything under tests/ to different R-Hub images.
For a big release, can also run `R CMD check --as-cran --run-donttest --use-valgrind <development tarball>` on a Linux machine with Valgrind for a thorough stress test (will take very long).

In case of a major release with breaking functionality, or a subsequent minor release, update the removed functions list in `deprecated.R`.

## Make sure there are no performance regressions

**check performance**

`perf.R`:

```R
options(width=120)
library(microbenchmark)
library(policytree)
set.seed(42)

# depth <- 3
n <- 100
p <- 5
d <- 4
X.c <- matrix(rnorm(n * p), n, p)
X.d <- matrix(sample(10:15, n * p, replace = TRUE), n, p)
X <- cbind(X.c, X.d, rbinom(n, 1, 0.5))
Y <- matrix(rnorm(n * d), n, d)
b1 <- microbenchmark(
tree <- policy_tree(X, Y, depth = 3),
unit = "ms",
times = 15)
b2 <- microbenchmark(
pp <- predict(tree, X),
unit = "ms",
times = 50)

# depth <- 2
n <- 10000
p <- 5
d <- 4
X.c <- matrix(rnorm(n * p), n, p)
X.d <- matrix(sample(10:15, n * p, replace = TRUE), n, p)
X <- cbind(X.c, X.d, rbinom(n, 1, 0.5))
Y <- matrix(rnorm(n * d), n, d)
b3 <- microbenchmark(
tree <- policy_tree(X, Y, depth = 2),
unit = "ms",
times = 15)
b4 <- microbenchmark(
pp <- predict(tree, X),
unit = "ms",
times = 50)
b5 <- microbenchmark(
tree <- policy_tree(X, Y, depth = 2, split.step = 10),
unit = "ms",
times = 15)

print(b1)
print(b2)
print(b3)
print(b4)
print(b5)
```

**check memory usage**

```
> R -d "valgrind --tool=massif" -f memory.R
> ms_print massif.out.34681 &> ms_print.out
```

The plot should look smooth, and should not contain large spikes in memory consumption and should not change drastically between releases.

`memory1.R`:

```R
library(policytree)
set.seed(42)
depth <- 2
n <- 10000
p <- 2
d <- 4
X.c <- matrix(rnorm(n * p), n, p)
X.d <- matrix(sample(10:15, n * p, replace = TRUE), n, p)
X <- cbind(X.c, X.d, rbinom(n, 1, 0.5))
Y <- matrix(rnorm(n * d), n, d)
tree <- policy_tree(X, Y, depth = depth)
pp <- predict(tree, X)
```

`memory2.R`:

```R
library(policytree)
set.seed(42)
depth <- 3
n <- 100
p <- 5
d <- 4
X.c <- matrix(rnorm(n * p), n, p)
X.d <- matrix(sample(10:15, n * p, replace = TRUE), n, p)
X <- cbind(X.c, X.d, rbinom(n, 1, 0.5))
Y <- matrix(rnorm(n * d), n, d)
tree <- policy_tree(X, Y, depth = depth)
pp <- predict(tree, X)
```

## Previous performance test results

**1.2.2**

This is only a patch release dropping CXX in Makevars per latest CRAN guidelines.

**1.2.1**

This is only a patch release updating the printed leaf labels for hybrid policy tree. No C++ changes.

**1.2.0**

(Machine: Sherlock 12 cores/50G/R version 3.5.1)

No changes to C++, only added hybrid_policy_tree.

perf

```
> print(b1)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 3) 4564.033 4573.721 4588.593 4578.084 4597.359 4638.117 15
> print(b2)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 0.0253 0.025779 0.03207804 0.026279 0.02687 0.291661 50
> print(b3)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2) 67672.76 68403.61 68538.73 68570.28 68733.21 68991.17 15
> print(b4)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 0.680048 0.732875 0.7332164 0.7373515 0.742225 0.788218 50
> print(b5)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2, split.step = 10) 8937.686 8967.728 8998.685 9004.088 9023.854 9042.722 15
>
```

memory1

```
MB
54.87^ #
|@:::::::::::::::@:::::::::::::::::#:::::::::::::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
|@:::::::: ::::: @: ::: :::::::::::#:::::::::: ::::@:::::@::::@:::::@::::
0 +----------------------------------------------------------------------->Gi
0 93.02
```

memory2

```
MB
54.31^ #
| #
| #
| #::@@:::::::@::@:::::@:::::::::::::::::::::::::::::@:::::@:::::@:::::@
| #::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| #::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| #::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| #::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| #::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
| :#::@ :: ::: @: @:::::@::: ::::: ::::::::::::: :::::@:::::@:::::@:::::@
0 +----------------------------------------------------------------------->Gi
0 24.37
```

**1.1.0**

(Machine: Sherlock 12 cores/50G/R version 3.5.1)

Notable changes since previous version: faster predict(), split.step skips observations instead of unique values, add min.node.size.

perf

```
> print(b1)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 3) 4608.389 4611.736 4653.743 4613.628 4648.493 4907.877 15
> print(b2)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 0.024629 0.025264 0.03059398 0.0257975 0.026444 0.240188 50
> print(b3)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2) 66994.34 67054.26 67187.36 67148.37 67267.58 67598.78 15
> print(b4)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 0.617329 0.6959 0.6944944 0.7075835 0.717939 0.726327 50
> print(b5)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2, split.step = 10) 8226.363 8229.001 8260.271 8231.613 8263.748 8403.574 15
```

memory1

```
MB
54.24^##
|# :::::::::::::@@::::@@:@::::::::@:::@::::::::::::::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
|# : :::::::::::@ :: :@ :@:::: :::@:::@: ::::::::: ::@:::@:::::@::::@::::
0 +----------------------------------------------------------------------->Gi
0 89.42
```

memory2

```
MB
54.24^ #
| #
| #
| #::::::::::::::::::::::@@:::::::::@@::::::@::::::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
| #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|::#::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
|: #::::::: ::::::: ::::::@ ::: :::::@ ::::::@:: :::::@:::::@:::::@:::::@
0 +----------------------------------------------------------------------->Gi
0 23.95
```

**1.0**

(Machine: Sherlock 12 cores/50G/R version 3.5.1)

perf
```
> print(b1)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 3) 4097.366 4099.058 4137.154 4109.452 4167.606 4224.911 15
> print(b2)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 1.935283 1.964984 2.066606 1.984469 2.046768 3.191294 50
> print(b3)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2) 59961.76 60042.89 60110.17 60128.08 60154.42 60290.77 15
> print(b4)
Unit: milliseconds
expr min lq mean median uq max neval
pp <- predict(tree, X) 133.6293 135.0255 135.9349 135.4114 135.8708 157.6832 50
> print(b5)
Unit: milliseconds
expr min lq mean median uq max neval
tree <- policy_tree(X, Y, depth = 2, split.step = 10) 7905.692 7929.35 7974.073 7933.476 7984.098 8223.15 15
```

memory1

```
MB
54.24^#
|#:::@::::::@::::::::@:::@::@::::::::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
|#:::@::::::@::: ::: @:::@::@:: :: ::::::::::::@:::::@:::::@:::::@::::::@
0 +----------------------------------------------------------------------->Gi
0 87.41
```

memory2

```
MB
54.24^ #
| #
| #
| #:::::::@@:::::@@:::::::@@::::@::::::::::::::::::::::::@:::::::::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| #: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
| :#: ::: :@ : :::@ : :::: @ : : @:::::::: :: :: ::: : :::@: :: : ::@::::
0 +----------------------------------------------------------------------->Gi
0 23.11
```

0 comments on commit 3b8ca17

Please sign in to comment.