Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NVIDIA] Removing hardcoded contributions and adding NVIDIA installation instructions #15

Draft
wants to merge 54 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
8666789
DEBUGGING
Oct 20, 2020
befa362
DEBUGGING
Oct 20, 2020
7603ca4
removing redundant directory
Oct 20, 2020
f9696d5
client wrapper code
Oct 20, 2020
98ae93a
removing extraneous files
Oct 21, 2020
41c4881
adding installation option (non-cython)
Oct 21, 2020
dd69b24
fixing non-cython install
Oct 21, 2020
b012d6f
fix non-cython install
Oct 21, 2020
1118955
fixing noncython install
Oct 21, 2020
a7dcf69
fixing noncython install
Oct 21, 2020
27b13ee
updating client and server wrappers, implementing NVIDIA functions in…
Oct 22, 2020
b8bba38
deleting unnecessary build files
Oct 22, 2020
09bd9d3
removing test build files
Oct 22, 2020
1b456af
slight modifications
Oct 27, 2020
13e1e41
changes to crypto constants
Oct 27, 2020
c068ea3
server wrapper changes
Oct 27, 2020
e3105d1
adding test
Oct 27, 2020
c07a865
changing return parameters of host_modelaggregator
Oct 27, 2020
68a81ee
preparing for import
Oct 29, 2020
c33e914
preparing for import
Oct 29, 2020
1aa7407
Adding aggregation python tests
Nov 3, 2020
7a963e1
Updating aggregation test
Nov 3, 2020
26cf44b
Cleanup and fixes
podcastinator Nov 3, 2020
f4551d0
Free allocated memory
podcastinator Nov 3, 2020
1983997
Adding support for n dimensional arrays
Nov 4, 2020
48b7331
Merge branch 'master' of github.com:kvah/secure-aggregation
Nov 4, 2020
320970c
cleaning files
Nov 4, 2020
ea80e4d
removing unnecessary files
Nov 4, 2020
981e9a6
Fix c++ integration test
chester-leung Nov 5, 2020
5d5987b
pushing nvidia test functions
podcastinator Nov 5, 2020
b63fb8a
Merge branch 'master' of https://github.com/kvah/secure-aggregation
podcastinator Nov 5, 2020
3be5490
Adding flatbuffer serialization
podcastinator Nov 10, 2020
dc5a609
Adding Flatbuffer Serialization Changes
podcastinator Nov 10, 2020
d61c11b
debugging with fake enclave
podcastinator Nov 10, 2020
4d46f94
debugging custom code
podcastinator Nov 11, 2020
cb51ab9
optimization
podcastinator Nov 11, 2020
e4c74cd
Docker start scripts
podcastinator Nov 11, 2020
ac911be
Run enclave code as well
podcastinator Nov 11, 2020
069950a
Increase enclave size
podcastinator Nov 11, 2020
b7b40c7
Checkpoint
podcastinator Nov 12, 2020
a2aeda9
Debugging Protobuf Issues
Nov 12, 2020
529e4cd
Checkpoint
podcastinator Nov 19, 2020
f3a25d2
Merge pull request #1 from kvah/rishabh
podcastinator Nov 23, 2020
67525d4
adding .gitignore, removing autogenerated cython files
Nov 24, 2020
33aefd1
Adding most recent version of NVIDIA Functions
Dec 10, 2020
2f9cde1
contributions no longer hardcoded; adding nvidia installation instruc…
Dec 24, 2020
beea8f7
removing unnecessary files
Dec 24, 2020
ce28cbc
Update .gitignore
Dec 24, 2020
c15b763
Update .gitignore
Dec 24, 2020
dafc5cc
Update .gitignore
Dec 24, 2020
6efe069
pushing latest changes (debugging seg fault during aggregation)
Jan 4, 2021
7058748
Updating nvidia custom functions
Jan 4, 2021
08711d0
Delete cmake-3.15.6-Linux-x86_64.sh
Jan 4, 2021
0f52d73
Delete get-pip.py
Jan 4, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified README.md
100644 → 100755
Empty file.
139,410 changes: 139,410 additions & 0 deletions admin/cmake-3.15.6-Linux-x86_64.sh

Large diffs are not rendered by default.

23,704 changes: 23,704 additions & 0 deletions admin/get-pip.py

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions admin/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
./startup/fl_admin.sh
Binary file added admin/startup/clara_hci-3.1.0-py3-none-any.whl
Binary file not shown.
17 changes: 17 additions & 0 deletions admin/startup/client.crt
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-----BEGIN CERTIFICATE-----
MIICwDCCAaigAwIBAgIUauNRWLVOu7xqbEe5qRXMfkgplb4wDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAwwPZXhhbXBsZV9wcm9qZWN0MB4XDTIwMTExODE5NDEzM1oX
DTIxMTExMzE5NDEzM1owGjEYMBYGA1UEAwwPZW1haWxAaGVsbG8uY29tMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsCbJMZtbdSJCYUcpT8fJ372hf6ZH
ciP608eOmu24GoaxW8fcLsd4bQCcoSUtt4EmQJCZrM/Jf9wsRMl/p7Mg9eGJHgA6
Hp44eGTiLjs6iCgkiC86a22HWgTnInD5Ci9oV0YqpW65AxEMU6hWd1g62WrM4Qgy
surzRNqYDAxrXDav+SmZP1K29tMtsVTU3QTKK5mj986L8Usy3MgFvYX5TLB1E8UR
gkjyXgZ0+HpyaK9Wd8ObkHHCYhSPjBVJo8BE+AX52iU6JAaxsGJJLBMNZbvfwtbh
zauMUGOZGyM+rw3wBW1loCx8dufsRzBtgdqe/lFbzB1P/u6PRw1ULeDYLwIDAQAB
MA0GCSqGSIb3DQEBCwUAA4IBAQB47tJ0ItbVTdGhHRS3aQD8xgd3azAPUC9jgZ0+
+Ks4QXialvsIz7CWWJpvpCEcdWNr4eKyUeMzKkWXVpwwwxIC4Ha4vENeXyGaXo+J
KtpTCyPKS8QfuPfq7SQwE+E+dCb3dLx35nAHa36PWt1wf76e/+17wWf5yePj66P9
Sf019TudkpuripuGlsXSYDFk6vkzoomZnJpWdBX8NCS0Tqi25K81GAMlR+akVCZZ
homoa7j5UW5Lju6xbbp8AGuUUbd5EyUB9Ejp062QGapDhkk48TCVS/U6nJsuZCjd
q21ca5rTOeEE2+xu5Rgq78eCqKGgX5/kFXGO6KpVIa+7Ut/B
-----END CERTIFICATE-----
27 changes: 27 additions & 0 deletions admin/startup/client.key
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAsCbJMZtbdSJCYUcpT8fJ372hf6ZHciP608eOmu24GoaxW8fc
Lsd4bQCcoSUtt4EmQJCZrM/Jf9wsRMl/p7Mg9eGJHgA6Hp44eGTiLjs6iCgkiC86
a22HWgTnInD5Ci9oV0YqpW65AxEMU6hWd1g62WrM4QgysurzRNqYDAxrXDav+SmZ
P1K29tMtsVTU3QTKK5mj986L8Usy3MgFvYX5TLB1E8URgkjyXgZ0+HpyaK9Wd8Ob
kHHCYhSPjBVJo8BE+AX52iU6JAaxsGJJLBMNZbvfwtbhzauMUGOZGyM+rw3wBW1l
oCx8dufsRzBtgdqe/lFbzB1P/u6PRw1ULeDYLwIDAQABAoIBAEdz9DjDJB4nf3Wm
QAO0rxaq/U+iS2jpX114nUgLFZrs+BSaxZ3UG+1bSi2zvw+ou7OE9FE9GCddUDe2
5KlJ6BzVF6GrxeeVlqJ/iZ/nNp70aZ00SyY4j6bcRkWaPer7v/9m3vWsLDAqQi3N
IwXWLEj1IZZGlQouc++CHBsHDqWgfLUwtHwTA83V8lIcJNlBGPo3RbW1/FGMkx+v
XoTAoI9VZIodAIL2FXakOkeVzEqWOx2NCbpam3Qys8eJODcTYPHALeFSsD5oH0PM
MOuT0yRlR8Z081eyXHa1fObxD73Kclx85RarcF3+L+en+tTI8jlhyV5Oyn/vcX51
FhMgkgECgYEA4en05jGfL+Z6/uF/jr3kWZhMs040Heioe+k1A+nlRygtoJw7QREg
A78aWViqobmJYFh2KAoOT6SXhGXK2Hv+FIc3eVwIkN9RLgutPyLBHILD2w1d6CZ7
jzq0fxSLpgR05jt46NA5DRtBiIyf4UDzWT8BxHwdUhMrX5c1QUBA56ECgYEAx5xJ
8i3oiA6h/ArvyQkdB0729ymu5xDye1wDzywnTv55PWYJIlWrqbPcTlsrYIMOy3u/
ZgyMYH5VmZKi4o25LOysm7/5hDPBySv0Ph3ruQgFOpmdtJTf1ANAv07zhC9kQ+Ls
Oq29mBkGeir3I741ja3ci63BMoz8aZAx24kGbc8CgYEAijj9EELYJRSduIj75Lgm
mPLoKuT0WkBZpcbgGw7mq4zXrecpEfyKXAB2/KeVbvH4LayyTnkonG4WDnnBjIcA
xEqfFG/NBea84rFeRLzc7rbLYqFhDYNOIipG+WH2t5MOezd0GO4dmflNNWDlCYsV
2UsFp9RQlEEKTqhTeyL3E0ECgYB6rW0s8FYXOJcJZldUgMtccIWMBgw+asX8VuLo
csR94hv1Rj6a4XgtRiNB7PpNyH5RRO6u80tbConlZp/BCUUjfRngZy4cuAqDbwYf
Hxh+kZVVHJA2KcEedhlW+ljjhpTivBZfQeRiFA9NChb12rLjLlV7awBv5wJChqJ9
wEs5NQKBgQCj1MS2uvH9KAeFTd5bLUSmuhUhC0YrpBHtOxCU9TvDVy/f3G5Rgv0/
Qe405bfyFPLL0XI9ylvL5I77QqdBmsKTsynJpjoFjYlIJSMTvaTn6HzUvzL/JtrU
RE0Zqn8QaR7VNIEnAs5+YoGpqeiLePFQyGCHpcDCr35HDbwThubcoA==
-----END RSA PRIVATE KEY-----
12 changes: 12 additions & 0 deletions admin/startup/docker.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# docker run script for FL admin
# to use host network, use line below
NETARG="--net=host"
# Admin clients do not need to open ports, so the following line is not needed.
#NETARG="-p 8003:8003"
DOCKER_IMAGE=clara-test
# DOCKER_IMAGE=clara-test
echo "Starting docker with $DOCKER_IMAGE"
docker run --rm -it --name=fladmin -v $DIR/..:/workspace/ -w /workspace/ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 $NETARG $DOCKER_IMAGE /bin/bash
4 changes: 4 additions & 0 deletions admin/startup/fl_admin.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
mkdir -p $DIR/../transfer
python -m dlmed.hci.tools.admin --host localhost --port 8003 --prompt "> " --with_file_transfer --upload_dir=$DIR/../transfer --download_dir=$DIR/../transfer --with_shell --with_login --with_ssl --cred_type cert --ca_cert=$DIR/rootCA.pem --client_cert=$DIR/client.crt --client_key=$DIR/client.key
20 changes: 20 additions & 0 deletions admin/startup/readme.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
*********************************
Admin Client package
*********************************
The package includes the following files:
readme.txt
rootCA.pem
client.crt
client.key
fl_admin.sh
fl_admin.bat
clara_hci-3.1.0-py3-none-any.whl

Please install the clara_hci-3.1.0-py3-none-any.whl file by 'python3 -m pip install.' This will install a set of Python codes
in your environment. After installation, you can run the fl_admin.sh or fl_admin.bat file to start communicating to the admin server.

The rootCA.pem file is pointed by "ca_cert" in fl_admin.sh/fl_admin.bat. If you plan to move/copy it to a different place,
you will need to modify fl_admin.sh. The same applies to the other two files, client.crt and client.key.

The email in your submission to participate this Federated Learning project is embedded in the CN field of client
certificate, which uniquely identifies the participant. As such, please safeguard its private key, client.key.
19 changes: 19 additions & 0 deletions admin/startup/rootCA.pem
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDEjCCAfqgAwIBAgIUTdQtq75ym/mUZIqnDiCq0yhZQegwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAwwPZXhhbXBsZV9wcm9qZWN0MB4XDTIwMTExODE5NDEzM1oX
DTIxMTExMzE5NDEzM1owGjEYMBYGA1UEAwwPZXhhbXBsZV9wcm9qZWN0MIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwI7fi3Eb2AG63ISJ6veJfCmwkh9c
pljxx+GOZyj+895xeyWYUdrsIRnVBYW6Yz+bydP3y6xOjBMnlTLZobv5SgldsieX
nTpxlyWutQzdjLaM9pgB+yY3x4yN3JIZxwn25poocW5lp6YwEyoMnO6d1oqScVEv
eBqSE/cDp/uzUkYRdfZZz+Gd1ByK/szs8rxSclYcwy9C9O2/qptETlSCoVdIlUG6
6nOk9oog8JlfzZIgJfUUOJw4An7Xvj4Wpw3kE23D7qjWlOeppvSAOXmOgZYY90dX
2MrAiP/9/jsPcIKComJ9nTlG2sFnqSGi6NLPdsuDG4IYWcIvx3+d0q8xvwIDAQAB
o1AwTjAdBgNVHQ4EFgQUS9j4lEv+zh4iqk6S6WdlcTcAqWowHwYDVR0jBBgwFoAU
S9j4lEv+zh4iqk6S6WdlcTcAqWowDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsF
AAOCAQEAMBwHYP7uO2TQJZecLEqPaEQ5+NyZa2gLtN7lkl5w9OyJiX2AI86ml5Ns
ioqPM/g40kmLoqnkafQopwv6VZXZJtO012QQCK8WXWcr+z8sqn+X4ZM2M6R6wzj/
XWBdBhAH5xv/6VIRXXyKIY7F99XXjRWTod1TU85UMkPkyU+AaRSUQHQRABpd5Ekf
zyh2DpnKlSSJTdcP/hUeDgd7heOd8gGEIG7e1Cx2UcZk6nwffkQI/qIBnQFdSXLI
DWY7Cib2HR5RpIlinCFPAAQi5iNRBHT2JgAdl0UPxmxjab04h3zILv2mgTq7iV2j
99CjGZbx0lgyPVxLMWg0aSx1/fwDNQ==
-----END CERTIFICATE-----
Binary file added admin/startup/signature.pkl
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
{
"batch_size": 1,
"DATA_LIST_KEY": "validation",

"pre_transforms":
[
{
"name": "LoadNifti",
"args": {
"fields": "image"
}
},
{
"name": "ConvertToChannelsFirst",
"args": {
"fields": "image"
}
},
{
"name": "ScaleIntensityRange",
"args": {
"fields": "image",
"a_min": -57,
"a_max": 164,
"b_min": 0.0,
"b_max": 1.0,
"clip": true
}
}
],

"post_transforms":
[
{
"name": "ArgmaxAcrossChannels",
"args": {
"fields": "model"
}
},
{
"name": "SplitBasedOnLabel",
"args": {
"field": "model",
"label_names": [
"pred_class0",
"pred_class1"
]
}
},
{
"name": "CopyProperties",
"args": {
"fields": ["pred_class0", "pred_class1", "model"],
"from_field": "image",
"properties": ["affine"]
}
}
],

"writers":
[
{
"name": "WriteNifti",
"args": {
"field": "model",
"dtype": "uint8",
"write_path": "{MMAR_EVAL_OUTPUT_PATH}"
}
},
{
"name": "WriteNifti",
"args":
{
"field": "pred_class0",
"dtype": "uint8",
"write_path": "{MMAR_EVAL_OUTPUT_PATH}"
}
},
{
"name": "WriteNifti",
"args":
{
"field": "pred_class1",
"dtype": "uint8",
"write_path": "{MMAR_EVAL_OUTPUT_PATH}"
}
}
],

"label_transforms":
[
{
"name": "LoadNifti",
"args": {
"fields": "label"
}
},
{
"name": "ConvertToChannelsFirst",
"args": {
"fields": "label"
}
},
{
"name": "SplitBasedOnLabel",
"args": {
"field": "label",
"label_names": [
"label_class0",
"label_class1"
]
}
}
],

"val_metrics":
[
{
"name": "ComputeAverageDice",
"args": {
"name": "mean_dice",
"field": "pred_class1",
"label_field": "label_class1",
"report_path": "{MMAR_EVAL_OUTPUT_PATH}"
}
}
],

"inferer":
{
"name": "TFScanWindowInferer",
"args": {
"roi_size": [160, 160, 160]
}
},

"model_loader":
{
"name": "CheckpointLoader",
"args": {
"checkpoint_dir": "{MMAR_CKPT_DIR}"
}
}
}
58 changes: 58 additions & 0 deletions admin/transfer/custom_ct_spleen/config/config_fed_client.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"servers": [
{
"name": "spleen_segmentation",
"service": {
"target": "localhost:8002",
"options": [
["grpc.max_send_message_length", 1000000000],
["grpc.max_receive_message_length", 1000000000]
]
}
}
],
"client": {
"local_epochs": 1,
"steps_aggregation": 0,
"exclude_vars": "(Adam|beta.*power|global_step)",
"cross_site_validate": false,
"privacy": {
},
"pre_processors": [
{
"name": "IteratorNumberGenerator",
"args": {
"field": "n_iter"
}
},
{
"path": "custom_encryptor.CustomModelEncryptor",
"args": {}
}
],
"post_processors": [
{
"path": "custom_encryptor.CustomModelDecryptor",
"args": {}
}
],
"req_processors": [
{
"name": "ClientStatusProcessor"
},
{
"name": "ValidateRequestProcessor"
}
],
"model_reader_writer": {
"name": "TFModelReaderWriter"
},
"model_validator": {
"name": "TFModelValidator"
},
"retry_timeout": 30,
"ssl_private_key": "resources/certs/client1.key",
"ssl_cert": "resources/certs/client1.crt",
"ssl_root_cert": "resources/certs/rootCA.pem"
}
}
57 changes: 57 additions & 0 deletions admin/transfer/custom_ct_spleen/config/config_fed_server.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"servers": [
{
"name": "spleen_segmentation",
"service": {
"target": "localhost:8002",
"options": [
["grpc.max_send_message_length", 1000000000],
["grpc.max_receive_message_length", 1000000000]
]
},
"max_num_clients": 100,
"min_num_clients": 1,
"wait_after_min_clients": 10,
"heart_beat_timeout": 600,
"start_round": 0,
"num_rounds": 10,
"exclude_vars": "(Adam|beta.*power|global_step)",
"num_server_workers": 4
}
],
"aggregator":
{
"path": "custom_aggregator.CustomModelAggregator",
"args": {
"exclude_vars": "(Adam|beta.*power|global_step)"
}
},
"pre_processors": [],
"post_processors": [],
"model_saver":
{
"path": "custom_model_saver.CustomModelSaver",
"args": {
"exclude_vars": "(Adam|beta.*power|global_step)"
}
},
"admin_cmd_modules": [
{
"name": "TrainingCommandModule"
},
{
"name": "ValidationCommandModule"
},
{
"name": "ShellCommandModule"
},
{
"name": "SystemCommandModule"
}
],
"result_processors": [
{
"name": "ValidateResultProcessor"
}
]
}
Loading