Restructure public/private headers

[broskoTT]

Motivation

There was an issue with tt_metal consuming umd, which was surfaced after #277.
The problem is that both tt-metal and umd have common dir, and that metal added umd's root dir to their include path. So in tt_metal, includes were ambiguous. This was solved by tenstorrent/tt-metal#15271

A follow up from that effort was that it wasn't clear what headers in umd are supposed to be publicly exposed. This issue is to track that work.

Initial change on clearing this up is being done in #322 .

Follow up changes might include renaming directories, to follow clearer structure of API headers and private implementation.
Details to be added in comment thread.

Suggested organization

@afuller-TT 's opinionated suggestion:
The root of the repo typically contains bookkeeping: Git files, lint configs, docs, etc. The code of a project should be immediately discoverable and clearly organized, but not tangled in with the peripheral files.

Something like this:

/
    .git-stuff
    .foo-stuff
    cmake/
    docs/
    src/
        api/
        src/
            firmware/
                api/
                src/
                test/
             moduleA/
                api/
                src/
                test/
        test/

The premise being that any node in the tree has a CMakeLists.txt in its root and can be relocated anywhere else and it remain consistent. It clearly defines its public API, contains its implementation, and holds the tests for that level of abstraction. This continues like turtles all the way down. moduleA in this example is also a self contained entity with unit tests that validate moduleA, and a defined external interface. It's clearly an implementation detail of the upper scope, and if one day it's deemed useful at a larger scope it could be relocated with a simple mv command and one add_subdirectory removed/added to relink it in its new home.

[broskoTT]

@afuller-TT I would say though that we have a single "module" which is currently called "device", but we can view it as "umd" module.
Would this simpler structure be sufficient?

/
    .git-stuff
    .foo-stuff
    cmake/
    docs/
    src/
        api/
        umd/
            firmware/ -> this one is just copied from another place, we don't intend on changing it.
                src/
            src/           -> holds private header files and all .cpp files
            test/          -> tests which know internals of umd
        test/              -> tests using only headers from api/, can also be viewed as "programming/api examples"

[afuller-TT]

@broskoTT I like your distinction between blackbox tests and whitebox tests. The blackbox tests in particular are useful for consumers to see how to interact with the module.

Bikeshedding a bit here, how do you like this flattening of the src/ so files share the same level of abstraction in the directory tree:

/
    .git-stuff
    .foo-stuff
    cmake/
    docs/
    src/
        api/
        src/
            firmware/ -> this one is just copied from another place, we don't intend on changing it.
                api/
                src/
                    test/
                test/
            .                -> holds private header files and all .cpp files
            test/          -> tests which know internals of umd
        test/              -> tests using only headers from api/, can also be viewed as "programming/api examples"

Essentially every 'module' (here we're only discussing the top-level UMD, currently named 'device', but the pattern should apply to anything at any scope), would consist of:

1. A directory that makes its name clear. For top-level, this is the name of the repo, so we push it down into a src to separate out the Git repo boilerplate, and which is a common directory name for people to understand. Other modules can just be their name like firmware above.
2. Within this directory we have api/ src/ and test/. The API contains the interface to the module in question. test/ contains the API-level tests (aka: black box testing, programming examples). src/ contains the implementation details. On a leaf node this will simply be the .cpp, .h, and a test/ for whitebox testing.
3. In larger modules we may break up sub sections of the implementation details and form a module of the same structure, nested under src/ that could trivially be hoisted out if that piece of functionality were valuable in a wider scope.

[broskoTT]

@afuller-TT I did read your points, but I still can't wrap my head around for root/src/src/ to be okay (the double src is the problem).

"files share the same level of abstraction in the directory tree"
Can you explain this further?
Were you talking about how includes would look like? In general how would includes look like internally and externally, I think this might drive us the right direction.

For example, in your suggestion, files would be:
root/src/api/public.h
root/src/src/private.h
root/src/src/internal.cpp
--- include "api/public.h"
--- include "src/private.h"
external.cpp
--- include "umd/src/api/public.h"

I'm sure I'm wrong, not sure why or where though.