Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C API用のzipファイルからmodelディレクトリをなくし、別でmodelダウンロード可能にする #603

Merged
merged 25 commits into from
Oct 13, 2023
Merged

C API用のzipファイルからmodelディレクトリをなくし、別でmodelダウンロード可能にする #603

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
57212bb
modelのzipファイルをアップロード
Hiroshiba Sep 9, 2023
0ef03a6
スクリプト版のダウンローダを消去する
Hiroshiba Sep 9, 2023
a6826cd
Merge branch 'スクリプト版のダウンローダを消去する' into コアのmodelディレクトリを分離
Hiroshiba Sep 9, 2023
2359609
ダウンローダーの更新
Hiroshiba Sep 9, 2023
887d615
modelディレクトリができてしまった
Hiroshiba Sep 9, 2023
035d3b1
製品版のバージョンを変更
Hiroshiba Sep 9, 2023
5ee41c4
Merge branch 'main' into コアのmodelディレクトリを分離
Hiroshiba Sep 12, 2023
4548853
github.event.inputs→inputs
Hiroshiba Sep 12, 2023
74ae74e
download_testをデフォルトでprerelease-latestが使われるように変更
Hiroshiba Sep 12, 2023
2e93f10
Merge remote-tracking branch 'upstream/main' into コアのmodelディレクトリを分離
Hiroshiba Oct 10, 2023
7889fa6
Merge remote-tracking branch 'upstream/main' into コアのmodelディレクトリを分離
Hiroshiba Oct 11, 2023
d5c370f
0.15.0-preview.2
Hiroshiba Oct 11, 2023
a948eb5
Merge remote-tracking branch 'upstream/main' into コアのmodelディレクトリを分離
Hiroshiba Oct 12, 2023
936ad70
修正漏れ
Hiroshiba Oct 12, 2023
ee48bcb
cargo run -p download -- -h
Hiroshiba Oct 12, 2023
d014d31
Apply suggestions from code review
Hiroshiba Oct 12, 2023
6bc9d62
変なとこが変わってた
Hiroshiba Oct 12, 2023
cee87ce
modelディレクトリのコピーが不要
Hiroshiba Oct 12, 2023
95c8dcf
update VOICEVOX_FAT_RESOURCE_VERSION
Hiroshiba Oct 12, 2023
c504620
min指定時にmodelディレクトリは無いはず
Hiroshiba Oct 12, 2023
cbd3e5b
Merge remote-tracking branch 'upstream/main' into コアのmodelディレクトリを分離
Hiroshiba Oct 13, 2023
4b755bc
--core_repo ${{ github.repository }}
Hiroshiba Oct 13, 2023
ee013d9
ミス
Hiroshiba Oct 13, 2023
62b5584
&
Hiroshiba Oct 13, 2023
ed5c8a6
Update .github/workflows/download_test.yml
Hiroshiba Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 28 additions & 14 deletions .github/workflows/build_and_deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ on:

env:
VOICEVOX_RESOURCE_VERSION: "0.15.0-preview.1"
VOICEVOX_FAT_RESOURCE_VERSION: "0.15.0-preview.1"
VOICEVOX_FAT_RESOURCE_VERSION: "0.15.0-preview.2"
# releaseタグ名か、workflow_dispatchでのバージョン名か、'0.0.0'が入る
VERSION: ${{ github.event.release.tag_name || github.event.inputs.version || '0.0.0' }}
PRODUCTION_REPOSITORY_TAG: "0.15.0-preview.2" # 製品版のタグ名
PRODUCTION_REPOSITORY_TAG: "0.15.0-preview.3" # 製品版のタグ名
# 簡易テストとするかどうか。releaseとworkflow_dispatch以外は簡易テストとする
IS_SIMPLE_TEST: ${{ github.event_name != 'release' && github.event_name != 'workflow_dispatch' }}

Expand Down Expand Up @@ -235,19 +235,11 @@ jobs:
repository: VOICEVOX/voicevox_resource
ref: ${{ env.VOICEVOX_RESOURCE_VERSION }}
path: download/resource
- name: Checkout VOICEVOX FAT RESOURCE
if: github.event.inputs.is_production == 'true'
uses: actions/checkout@v3
with:
repository: VOICEVOX/voicevox_fat_resource
ref: ${{ env.VOICEVOX_FAT_RESOURCE_VERSION }}
path: download/fat_resource
- name: Raplace resource
if: github.event.inputs.is_production == 'true'
shell: bash
run: |
mv -f download/resource/core/README.md ./README.md
rm -r ./model; mv download/fat_resource/core/model ./model
- name: Install cargo-binstall
uses: taiki-e/install-action@cargo-binstall
- name: Install cargo-edit
Expand Down Expand Up @@ -318,7 +310,7 @@ jobs:
cd artifact
7z a "../${{ env.ASSET_NAME }}.zip" "${{ env.ASSET_NAME }}"
- name: Upload to Release
if: env.VERSION != '0.0.0' && env.SKIP_UPLOADING_RELEASE_ASSET == '0' && !contains(matrix.target, 'ios')
if: env.VERSION != '0.0.0' && !contains(matrix.target, 'ios')
uses: softprops/action-gh-release@v1
with:
prerelease: true
Expand Down Expand Up @@ -381,7 +373,7 @@ jobs:
cd artifact/${{ env.ASSET_NAME }}
7z a "../../${{ env.ASSET_NAME }}.zip" "voicevox_core.xcframework"
- name: Upload to Release
if: env.VERSION != '0.0.0' && env.SKIP_UPLOADING_RELEASE_ASSET == '0'
if: env.VERSION != '0.0.0'
uses: softprops/action-gh-release@v1
with:
prerelease: true
Expand All @@ -390,16 +382,38 @@ jobs:
${{ env.ASSET_NAME }}.zip
target_commitish: ${{ github.sha }}

deploy_downloader:
deploy_model:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Checkout VOICEVOX FAT RESOURCE
if: github.event.inputs.is_production == 'true'
uses: actions/checkout@v3
with:
repository: VOICEVOX/voicevox_fat_resource
ref: ${{ env.VOICEVOX_FAT_RESOURCE_VERSION }}
path: download/fat_resource
- name: Raplace resource
if: github.event.inputs.is_production == 'true'
shell: bash
run: |
rm -r ./model; mv download/fat_resource/core/model ./model
- name: Set ASSET_NAME env var
run: echo "ASSET_NAME=model-${{ env.VERSION }}" >> "$GITHUB_ENV"
- name: Create artifact
run: |
mkdir "artifact"
mv model "artifact/${{ env.ASSET_NAME }}/"
- name: Archive artifact
run: |
cd artifact
7z a "../${{ env.ASSET_NAME }}.zip" "${{ env.ASSET_NAME }}"
- name: Upload to Release
if: env.VERSION != '0.0.0'
uses: softprops/action-gh-release@v1
with:
prerelease: true
tag_name: ${{ env.VERSION }}
files: |-
scripts/downloads/*
${{ env.ASSET_NAME }}.zip
target_commitish: ${{ github.sha }}
115 changes: 0 additions & 115 deletions .github/workflows/download_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ on:
paths:
- "Cargo.*"
- "crates/download/**"
- "scripts/downloads/*"
- ".github/workflows/download_test.yml"
defaults:
run:
Expand Down Expand Up @@ -168,120 +167,6 @@ jobs:
cufft64_*.dll
curand64_*.dll
open_jtalk_dic_utf_8-1.11
- name: 通常ダウンロード
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh
download_dir: voicevox_core
check_items: |
libvoicevox_core.so
model/metas.json
open_jtalk_dic_utf_8-1.11
README.txt
check_not_exists_items: |
*directml*
*cuda*
*cudnn*
*onnxruntime*providers*
*eula*
*nvidia*
*cufft*
*curand*
- name: CPUArch指定
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh --cpu-arch x64
download_dir: voicevox_core
check_items: |
libvoicevox_core.so
model/metas.json
open_jtalk_dic_utf_8-1.11
README.txt
check_not_exists_items: |
*directml*
*cuda*
*cudnn*
*onnxruntime*providers*
*eula*
*nvidia*
*cufft*
*curand*
- name: output先指定ダウンロード
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh --output other_output
download_dir: other_output
check_items: |
libvoicevox_core.so
model/metas.json
open_jtalk_dic_utf_8-1.11
README.txt
check_not_exists_items: |
*directml*
*cuda*
*cudnn*
*onnxruntime*providers*
*eula*
*nvidia*
*cufft*
*curand*
- name: min option確認
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh --min
download_dir: voicevox_core
check_items: |
libvoicevox_core.so
model/metas.json
README.txt
check_not_exists_items: |
*directml*
*cuda*
*cudnn*
*onnxruntime*providers*
*eula*
*nvidia*
*cufft*
*curand*
open_jtalk_dic_utf_8-1.11
- name: cuda option確認
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh --device cuda
download_dir: voicevox_core
check_items: |
libvoicevox_core.so
model/metas.json
open_jtalk_dic_utf_8-1.11
README.txt
EULA.txt
NVIDIA_SLA_cuDNN_Support.txt
libcublas.so.*
libcublasLt.so.*
libcudnn.so.*
libcudnn_adv_infer.so.*
libcudnn_cnn_infer.so.*
libcudnn_ops_infer.so.*
libcufft.so.*
libcurand.so.*
check_not_exists_items: |
*directml*
- name: cudaかつmin option確認
os: ubuntu-latest
download_command: ./scripts/downloads/download.sh --device cuda --min
download_dir: voicevox_core
check_items: |
libvoicevox_core.so
model/metas.json
README.txt
check_not_exists_items: |
*directml*
EULA.txt
NVIDIA_SLA_cuDNN_Support.txt
libcublas.so.*
libcublasLt.so.*
libcudnn.so.*
libcudnn_adv_infer.so.*
libcudnn_cnn_infer.so.*
libcudnn_ops_infer.so.*
libcufft.so.*
libcurand.so.*
open_jtalk_dic_utf_8-1.11
runs-on: ${{ matrix.os }}
name: ${{ matrix.name }}-${{ matrix.os }}
env:
Expand Down
25 changes: 0 additions & 25 deletions .github/workflows/psscriptanalyzer.yml
View file
Open in desktop

This file was deleted.

6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,12 @@ DLL用のヘッダファイルは [crates/voicevox\_core\_c\_api/include/voicevo
cargo test
```

## ダウンローダーの実行

```bash
cargo run -p download
```
Hiroshiba marked this conversation as resolved.
Show resolved Hide resolved

## ヘッダファイルの更新

```bash
Expand Down
14 changes: 14 additions & 0 deletions crates/download/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,11 @@ async fn main() -> anyhow::Result<()> {
})
.await?;

let model = find_gh_asset(octocrab, CORE_REPO_NAME, &version, |tag| {
Hiroshiba marked this conversation as resolved.
Show resolved Hide resolved
format!("model-{tag}.zip")
})
.await?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

find_gh_assetをfind_gh_assetsにすればアクセス回数を一つ減らせそうではありますね。以下のようにまですべきかどうかはともかくとして。

-    let core = find_gh_asset(octocrab, CORE_REPO_NAME, &version, |tag| {
+    let [core, model] = find_gh_assets(octocrab, CORE_REPO_NAME, &version, |tag| {
         let device = match (os, device) {
             (Os::Linux, Device::Cuda) => "gpu",
             (_, device) => device.into(),
         };
-        format!("{CORE_REPO_NAME}-{os}-{cpu_arch}-{device}-{tag}.zip")
-    })
-    .await?;
-
-    let model = find_gh_asset(octocrab, CORE_REPO_NAME, &version, |tag| {
-        format!("model-{tag}.zip")
+        [
+            format!("{CORE_REPO_NAME}-{os}-{cpu_arch}-{device}-{tag}.zip"),
+            format!("model-{tag}.zip"),
+        ]
     })
     .await?;

     let additional_libraries = OptionFuture::from((device != Device::Cpu).then(|| {
-        find_gh_asset(
+        find_gh_assets(
             octocrab,
             ADDITIONAL_LIBRARIES_REPO_NAME,
             &additional_libraries_version,
             |_| {
                 let device = match device {
                     Device::Cpu => unreachable!(),
                     Device::Cuda => "CUDA",
                     Device::Directml => "DirectML",
                 };
-                format!("{device}-{os}-{cpu_arch}.zip")
+                [format!("{device}-{os}-{cpu_arch}.zip")]
             },
         )
     }))
     .await
-    .transpose()?;
+    .transpose()?
+    .map(|[asset]| asset);
-async fn find_gh_asset(
+async fn find_gh_assets<const N: usize>(
     octocrab: &Arc<Octocrab>,
     repo: &str,
     git_tag_or_latest: &str,
-    asset_name: impl FnOnce(&str) -> String,
-) -> anyhow::Result<GhAsset> {
+    asset_names: impl FnOnce(&str) -> [String; N],
+) -> anyhow::Result<[GhAsset; N]> {
     let Release {
         html_url,
         tag_name,
         assets,
         ..
     } = {
         let repos = octocrab.repos(ORGANIZATION_NAME, repo);
         let releases = repos.releases();
         match git_tag_or_latest {
             "latest" => releases.get_latest().await,
             tag => releases.get_by_tag(tag).await,
         }?
     };

-    let asset_name = asset_name(&tag_name);
-    let Asset { id, name, size, .. } = assets
-        .into_iter()
-        .find(|Asset { name, .. }| *name == asset_name)
-        .with_context(|| format!("Could not find {asset_name:?} in {html_url}"))?;
+    return asset_names(&tag_name).try_map_(|asset_name| {
+        let Asset { id, name, size, .. } = assets
+            .iter()
+            .find(|Asset { name, .. }| *name == asset_name)
+            .with_context(|| format!("Could not find {asset_name:?} in {html_url}"))?;
+
+        Ok(GhAsset {
+            octocrab: octocrab.clone(),
+            repo: repo.to_owned(),
+            tag: tag_name.clone(),
+            id: *id,
+            name: name.clone(),
+            size: *size as _,
+        })
+    });

-    Ok(GhAsset {
-        octocrab: octocrab.clone(),
-        repo: repo.to_owned(),
-        tag: tag_name,
-        id,
-        name,
-        size: size as _,
-    })
+    #[easy_ext::ext]
+    impl<T, const N: usize> [T; N] {
+        fn try_map_<F, O, E>(self, f: F) -> Result<[O; N], E>
+        where
+            F: FnMut(T) -> Result<O, E>,
+        {
+            let vec = self.into_iter().map(f).collect::<Result<Vec<_>, _>>()?;
+            Ok(vec
+                .try_into()
+                .unwrap_or_else(|_| unreachable!("the lengths should be same")))
+        }
+    }
 }

Copy link
Member Author

@Hiroshiba Hiroshiba Oct 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

すみません、まとめた方が良さそうというのはわかるのですが、ちょっと難しすぎそうなのでパスさせてください 🙇


let additional_libraries = OptionFuture::from((device != Device::Cpu).then(|| {
find_gh_asset(
octocrab,
Expand Down Expand Up @@ -186,6 +191,15 @@ async fn main() -> anyhow::Result<()> {
)?);

if !min {
let model_dir = output.join("model");
fs_err::create_dir(&model_dir)?;
tasks.spawn(download_and_extract_from_gh(
model,
Stripping::FirstDir,
&model_dir,
Hiroshiba marked this conversation as resolved.
Show resolved Hide resolved
&progresses,
)?);

if let Some(additional_libraries) = additional_libraries {
tasks.spawn(download_and_extract_from_gh(
additional_libraries,
Expand Down
Loading
Loading