-
Notifications
You must be signed in to change notification settings - Fork 292
System DT Meeting Notes 2019
Nathalie Chan King Choy edited this page Oct 1, 2019
·
9 revisions
- Logistics: When to meet at Linaro Connect? When is a good time to have regular meetings, how often, other people to involve, ... - Tomas/Nathalie
- Quick recap of the problems that System-Device Trees are solving - Tomas
- Proposed additions to specification to describe multiple CPU clusters - Stefano
- Proposed conventions to allocate resources to domains - Stefano
- Update on Lopper tool - Bruce
Arnd Bergmann, Arvind (Mentor), Benjamin Gaignard (ST), Bill Fletcher - Linaro, Bill Mills (TI), Bruce Ashfield, Clément Léger (Kalray), Dan Driscoll, Danut Milea, Ed Mooring, Etsam Anjum (MGC), felix, Iliad, Jan Kiszka, jeremy gilbert (Qualcomm), Joakim Bech, Loic Pallardy (ST), Maarten Koning, Manju kumar, Nathalie Chan King Choy, Stefano Stabellini, Steve McIntyre, Suman Anna (TI), Tomas Evensen, Tony McDowell (Xilinx), Trilok Soni
- Tomas: Challenge with System DT is that it's cross-functional, so no 1 single mailing list to discuss
- Want to start with some live discussions
- Will formalize proposal & take to mailing lists
- Anyone opposed to recording: No. Recording starts here.
- Intros by attendees
- Tomas: Quick intro on System DT
- Background:
- SoCs very heterogeneous w/ multiple CPU clusters
- System SW needs info about the system (registers in memory map, topologies, DDR allocation, etc.)
- Different personas involved
- HW architect
- System architect
- Info for each operating system
- Config info comes in at different times: Sometimes the config info is compiled in, sometimes loaded at boot, etc.
- Not just multiple CPUs
- other accelerators may need memory
- Execution levels
- Trustzone
- Today, how this is specified is ad hoc. Want 1 true source & tooling that can get the info to the right place.
- A lot of RTOS vendors started to do work with Device Trees, so a natural source
- Verification is also a really important part of it
- At Xilinx, had an XML file & tooling to give the info to Linux, baremetal, free RTOS
- Not standardized
- Need an industry standard. Device Tree is almost there.
- DT looks at 1 address space, what Linux sees or what Xen sees.
- Need to expand DT -> System DT
- It should do 2 things:
- Describe HW for all the different masters, or CPU clusters
- Need to add some things into devicetree.org spec & tooling
- Describe AMP config info: what goes where & domain
- Suggesting "chosen" section
- Want to be compatible with hypervisor. (Stefano a maintainer with Xen, so makes sense to align)
- Describe HW for all the different masters, or CPU clusters
- It should do 2 things:
- Lopper creates traditional DT based on System DT
- So, we don't have change the DT that Linux or U-Boot sees
- Maybe longer term, Linux may want to know about System DT
- System DT data flow (RTOS & Linux) diagram showing different personas & sources of info
- Can add new nodes, add attributes into node, suppress things
- System architect info can be added as snippet
- Want to change 1 part of system without editing everyone else's files, but could still be merged together later
- Showing some possible flows for RTOS/baremetal & Linux
- Example: System DT + domain info from system architect -> Linux DT using Lopper
- Flow diagram for Xen & QEMU
- Intent to get it to work with System DT in future
- What is shown with QEMU is Xilinx QEMU. Upstream does differently. Trying to get Xilinx method upstream.
- Domains & resource groups
- Is what you are sharing intended to be shared?
- Background:
- Questions for Tomas about vision?:
- No questions
- Tomas: We found a lot of ppl have solved similar problems, which makes it interesting
- Mentor has some tooling & syntax to specify allocations. Ahead of where we are in terms of the problems encountered. Looking forward to your feedback.
- Wind River using DT for VxWorks in different way
- How do you distinguish between in TZ & outside TZ?
- Really want this to be a universal solution as much as possible
- Face to Face at Linaro Connect
- Will have possibility for people to call in
- Wed afternoon or Thur morning
- Steve: Suggest Wed 3:30pm PDT for SystemDT but OK to move
- Who would be calling in from other time zones? Will have Europe
- Morning is boot session from LEDGE, maybe could swap? Wed morning is kernel & ppl need to be there.
- 9am-10:30am PDT Thur morning looks most promising. 16:00 UTC
- Steve will try to make these show up on public schedule
- Perhaps may have ~30 ppl
- Stefano: Representative example (not complete) of System DT that would be input to Lopper
- Output of Lopper could be used by Linux, or RTOS
- 1st set of changes: How to represent CPU clusters
- DT mandates only 1 CPU notes called "cpus"
- We want to represent A53 cluster & R5 cluster, which have different architectures
- Want to introduce concept of CPU cluster -> small change to spec, but deep impact to DT
- Keep in mind: Different CPU clusters can have different memory maps
- Example on Xilinx board where A53 memory map is different from R5 memory map: e.g. tcm_bus region
- Available at 2 addresses for R5 cluster (low address and high address)
- Purpose is to show different memory map for 2 different CPU clusters
- ranges-map is showing AMBA is accessible 1:1 & TCM bus mapping in A53, in R5 have additional entry for TCM
- These are changes that we need for System DT that can represent multiple heterogeneous CPUs
- Nothing about configuration shown here
- Domain section under "chosen": How to explain configuration
- Could be changed in SW or firmware
- What is configured to run where
- What privilege level on CPUs
- What shared memory we will have between different domains (area where 1 set of SW runs, e.g. R5 domain vs. A53 domain)
- We have a memory property that shows address & size for RAM region accessible from this domain
- CPU attribute is interesting b/c tells us where this domain is running (e.g. R5 cluster & CPU mask shows which CPU used to run this domain)
- Architecture-specific # can represent an architecture-specific config of the CPU
- e.g. ARM R5 can be lockstep or not lockstep and secure mode or normal mode
- Other key part is the "access" property
- Expressing in list of links what is accessible from this domain
- On Xilinx & other vendor boards, there is possibility to enforce isolation
- This expresses if this OpenAMP domain can access
- Example: Showing normal memory, TCM bus, reserved memory region for communicating with A53, IPI is Xilinx HW block for communication between A53 & R5 cluster
- Flag: provide extra info
- e.g. mailbox: Which channel, TX or RX
- Each vendor could express vendor-specific meaning for flag to say how mapping should be done
- Could then be translated into remoteproc configuration as used by Linux. Conceptually maps to the info we saw before. Could be generated by Lopper from System DT.
- Stefano has been getting inputs from different groups & starting to reach out wider.
Tomas, Nathalie, Stefano, Arnd, Maarten, Dan (Mentor), Grant, Ed, Vincent, Matthew, Kumar, Azzedine, Jeremy, Dan (Wind River), Loic, Erwan, Steve, Bruce, Mark, Joakim, Achin, Sudipto, Anders, Manju, Saravana, Arvind, Krzystof, Clement, Mark (Xilinx), Etsam, Waqas, Vivek, Bjorn, Rob, Trilokkumar, Etienne, Victor
Dive into the details of what we have done so far, get feedback, who is interested in engaging, ask questions, identify open issues to work on
- 2 parts
- What changes needed to spec
- Configuration
- Stefano: showed proposed changes to spec using example code
- Arnd: What about interrupts?
- Stefano: idea is to use interrupt extended property
- e.g. R5 cluster will have different GIC from A53 cluster
- Lopper will be able to chop out interrupt hierarchy that's not relevant to you CPU cluster
- Stefano: idea is to use interrupt extended property
- On use of term "ranges"
- Rob: We should not call it ranges b/c it's not ranges. Something map, but map has a defined meaning as well.
- Grant: If you put the full system tree to OS, will it still make sense? Do you need lopper? Want simplest case to work with System DT as given. Want additive & not transformative. Not opposed to changes to Linux. Would like there to be very high bar for proposal that breaks Linux (e.g. go to 2 step tool). Only if it's the only way to do it.
- Rob: We should minimize the processing that takes place to create a bootable DT
- Bjorn brought up that UART might require processing
- Stefano: Need 1 more hook for interrupts. Don't think 0 changes to Linux is possible.
- Arnd: Will depend on platform.
- Kumar: Why not be explicit, like interrupt extended property. We get in trouble with defaults. Being explicit removes question of wanting root cluster a certain way. "cpu" cluster has not kept up & is antiquated. To maintain compatibility, we should be more explicit in map property & then you don't have to worry about which cluster. Map applies to CPUs and which it applies to.
- Ed: The current system is broken for someone trying to use non-Linux OS on a heterogeneous system. 1/3 time of OpenAMP development is trying to get Linux DT to match what you're doing. There's a lot of breakage on the other side of the fence.
- Grant: Let's take it offline. This is awfully close. If a53 was just named cpus and in common case where there isn't a CPU map, it works. Let's go through some scenarios when it gets concrete. We know it's an issue.
- Tomas: System DT is not for everyone. You can skip it if you don't need it, and continue to use include files. Ideal if you could make it backward compatible.
- Grant: Would like to see it as core part of DT tooling. Don't want another tool off to the side.
- Tomas: Agree
- Azzedine: Does it mean we have action to define what mapping needs to be?
- Stefano: Yes
- Software configurations
- Chosen node -> SW configuration
- What kernel runs where
- Which resources supposed to & not supposed to use. No HW limitation in usage of the resources, it's SW
- Kumar & Stefano: on where to draw the line for what goes in "chosen"
- Stefano has Secure/Non-secure and R5 lockstep covered under chosen. Lockstep is strange b/c affects CPU configuration.
- Kumar: Don't think you should put info about how the HW works in chosen. Chosen is for configuration. HW map in secure should be natural DT.
- Stefano: Some things are in grey area - this is one. If we see System DT as static that comes from manufacturer in your box with your kit. The board in the box can be one or the other. It's the customer that will decide if lockstep or not.
- Kumar: Then every config option now has to move here. Don't think that's feasible. Either you have references to everything or duplicate everything. You'll have 1001 properties you'll have to configure this way.
- Kumar: Whether it's a CPU or a UART, it's just a sub part of a system and has properties & can have defaults that cause behavior to occur.
- Stefano: Will take this as a note to think more on it.
- Tomas: Somewhere all this info needs to be specified. Shouldn't be device mfrer, should be customer. We started with a YAML file & lopper can handle. We decided to continue using DT syntax b/c easier for the tool. We need to separate personas doing different things. Don't want to make someone change in many places for 1 thing. Should we have a separate format? That's a separate discussion.
- Kumar: There is a DT initiative around configuration. As we have schema info for the bindings, how to specify if property needs to be configured. Should user specify? Should it be expert mode? What is more convenient to user?
- To consider: OS or firmware as consumer?
- Chosen node -> SW configuration
- Tomas: Have added some real use cases around OpenAMP & realize that what, for e.g. Xilinx, ST, TI, need are different
- Should Linux be able to handle system DT without lopper? Is it viable? Do we need to change Linux?
- How to handle interrupt controller?
- Ranges map - should it be called something different
- Secure: Where to put the info - hw vs. configuration. Don't want to duplicate notes. Want SoC vendor to be able to describe the HW & say what you can do. What should input be? DT format, YAML
- Rob: Suggestion for process
- Better if we do implementation details on email review in device tree spec list
- In the call, stick to more high-level things like what do we want to solve
- Ranges
- Interrupts
- How do we assign devices
- Azzedine: Pick 1 topic at a time to focus on in the follow up meetings.
- Tomas: There are lots of ppl who are interested -> should recommend they join the device tree spec mailing list
- Stefano: Virtualization - memory property - address range
- Azzedine & Stefano to discuss offline