Progressive Enviroment Building

[aseaday]

I do some research on new kinds of model or problems. When I begin a new project, there is only limited packages or set-up I know in advance. After a while, I woul add more and more packages, data or code into the project.

Envd do a lot of work to create a environment defined well previously. It may only be available in some mature projects of a team. There is still a lot of work to be done If you are the owner of new projects and want to share it among your team. You need to remember or record what you do to the environment and update the envd manifest.

So I think we should work some good ideas to solve this problem. I name this problem as Progressive Enviroment Building. There is little effort needed by a data scientist. She/He just init a new envd environment and do her/his stuff, something will update his changes to the manifest. When her/his work is done. the manifest also become a ready-share configuration. Here is a example:

No Progressive Building Support

I decieded to develop a Graph toolkit to analyze the connection or flow among the DeFi contracts in the BlockChain. I have no idea about Blockchain or DeFi before.

1. I thought I need get data first, I found it in eth-etl: pip3 install ethereum-etl
2. In order to visualize some data results, I found Visdcc, and install it
3. There also were a lot of other toolkits or data needed to be download one by one.
   After a long time, I thought I could set up a environment for later use or sharing among team. So I began to write build.envd. But I had forget some steps I did. conda list -e could help it but it will bring a lot of deps. In a word, you had to recall and construct all the process on the day you decide to set up a envd.

Progressive Building Support

A idea is that user could build env during the development/exploration.

1. I inited a envd shell in progressive mode and loged into the container.
2. I thought I need get data first, I found it in eth-etl: pip3 install ethereum-etl, there may was a daemon in the shell would detect it and ask users:

Add the ethereum-etl 

3. There may were other data/big files hook to detect the changes and asked users to add the change to the manifest.
4. When I finished the first stage's goal, the manifest was also prepared well.

[gaocegege]

Thanks for raising this! Could you please share an example of it? For example, when you need to set up a new research project, what you will do with your dev env.

[aseaday]

Add example.

[gaocegege]

cc @kemingy @VoVAllen

[VoVAllen]

Thanks for the proposal. There're several way we could achieve this:

• Hack the shell to record all the command history and filter out the pip related command
• Include a custom binary, for example named as e, which will transparently forward commands, but also record the commands. User need to do e pip install ethereum-etl to explicitly tell envd to add this dependency to the build.envd

From a bigger picture, I think user should be able to add common-used packages to build.envd without any effort. Envd might needs to maintain a recipe index like brew. And user can easily contribute their recipe also, especially for those packages that are complicated to install.

WDYT?

[aseaday]

brew is a good idea. I could write a data analyze tools to detect the most popular packages of Github.
A develop toolkits for data scientist should be driven by data science!