-
Notifications
You must be signed in to change notification settings - Fork 292
System DT Meeting Notes 2019
Nathalie Chan King Choy edited this page Oct 31, 2019
·
9 revisions
Grant L. (Arm), Tomas E. (X), Benjamin G. (ST), Dan M. (WR), Don H. (L), Etsam A. (MGC), Loic P. (ST), Mark G. (TI), Mathieu P. (L), Rob H. (Arm), Steve McI. (L), Stefano S. (X), Poonam A. (NXP), Vincent G. (L), Joakim B. (L), Ed M. (X), Manju HM (X), Nathalie CKC (X)
- Stefano: Remove ranges from amba_rpu
- Stefano: In next couple weeks, look at GPIO controller extensions & write in detail & send to list so we can discuss.
- All: if you can think of simple solution to interrupt-map info duplication, let Stefano know & he will try to add to the example.
- Nathalie: Confirm with Kumar that Wed slot will work & send out meeting invitation for 4 weeks from now.
- Nathalie: Add Connect meeting attendees & send out info on how to subscribe to OpenAMP system-dt mailing list.
- Nathalie: At bottom of all system-dt-related emails, put how to join the list/get involved/learn more
- Mailing list
- Tomas: Use DT list? OpenAMP list? OpenAMP, then DT list?
- Grant: Since we're not quite sure what it's going to look like, until we have concrete proposal, then doing stuff on OpenAMP list is fine. Once have something concrete, then can move over. This mitigates concern for too much noise.
- Steve: Agree. Let's thrash out prototype before letting ppl pick it apart.
- Rob: Think only crickets will respond. Suggested device tree spec list b/c not a lot of traffic there. Pretty much everyone cares b/c everyone has complex systems. Any SoC has Trust Zone, system controller & those start to use DT.
- Tomas: Let's start on OpenAMP list. Want to include proprietary OS vendors in the discussion too. Once we converge, then can move over to DT list.
- Nathalie: Who should we have on the list? OpenAMP TSC list, DTE meeting attendees so far.
- Tomas: Add anyone who joined the Connect call. Email out details for how to join the list.
- Steve: Put bottom of any mail out to ppl info on how to join list & where to get info to get involved
- Tomas: Also include pointers to the wiki landing page, where we have the video from the intro presentation.
- Tomas: Use DT list? OpenAMP list? OpenAMP, then DT list?
- What works for a regular meeting slot and cadence?
- Tomas: Proposing once a month for cadence
- Steve, Grant: Seems about right
- Nathalie: What about the Wednesday slot in the weeks Steve isn't holding the DTE call, every 4 weeks?
- Tomas & Steve: OK
- No objections on the call
- Nathalie: Circle back with Kumar to confirm Wed slot works for the Zephyr folks
- Nathalie: Next call in 4 weeks on Wed, if Kumar/Zephyr don't have conflict
- Tomas: Proposing once a month for cadence
- Stefano: Update on what happened since Connect
- Stefano: Worked with Grant to get system DT to work in default case
- Grant: Regardless of the overall intent of system DT, want to see all the changes/bindings that go into making this work be fairly well defined for each individual component & they shouldn't be too interlinked.
- E.g. Address mapping: DT has always had assumption that address map starts at root node & develops from there. By using the address map, that changes the default & can fan out to different nodes in the tree. It's well-described, well-contained & can be implemented independently from the other system DT stuff.
- Don't think it's required for you to build a DT that will work with an old kernel, but would like it to be possible
- Think Stefano & I were able to get to that point
- Grant: Regardless of the overall intent of system DT, want to see all the changes/bindings that go into making this work be fairly well defined for each individual component & they shouldn't be too interlinked.
- Stefano: Based example on Xilinx Versal device tree. Changes minimal. Rob has been asking for a full example. Have a full example now that can be shown.
- Address map & CPU cluster still there
- Interrupt done with intermediary interrupt controller node. Still using the standard bindings, so don't need new bindings.
- New bus type (indirect bus - thanks, Grant!) so can put things that don't map into the root address space. If you have default CPU cluster that is Cortex A. Linux will boot & fall naturally into that address space. System DT will describe things that might not be accessible/visible (e.g. interrupt controller of cortex-R5, which is not visible to Cortex-A) -> Indirect bus will hide it from the Cortex-A. Wouldn't translate automatically in the parent address node, so Linux wouldn't try to access it.
- Thanks to Grant for example
- Stefano showed this example early draft system DTS file
- This time started with full device tree as example. Short example shown at Connect was artificial.
- A72s as default CPU node
- Has "cpus" name & doesn't have address map
- cpus_r5
- Has address map & compatible CPU cluster
- How do we build with R5 GIC? A72 GIC should be only visible & accessible from A72.
- Amba_rpu with indirect bus
- Grant: This node has a ranges property, which says that it maps to root node. Is that what you intended?
- Stefano: Wasn't sure - question mark.
- Grant: It should not have. Ranges should be removed.
- Amba_rpu with indirect bus
- Address ranges: Only update is this indrect bus
- Next, interrupts
- Refer to "interrupt-multiplex" in the example
- Will be parent of all devices, going to both gic
- "interrupt-map" property
- First three numbers on left match the device interrupt. The zeros -> ignoring those fields.
- e.g. 0x14 going to gic_a72
- Then numbers on right indicate where going to instance on A72 & could be different
- Down side is this is cumbersome & repeat every interrupt 3 times (at the device, second time for the A72, third time to say it's going to the R5)
- Grant: This could be somewhere that needs new binding. Moving into a new node does make sense b/c the device still has only 1 output line. Probably only need 1 cell to describe the input to the distributor node b/c there is no state associated (don't care if level or edge). Think only need the fields if it's a real interrupt controller - what you have here is probably just wire going to multiple GICs.
- Stefano: Yes, think just wire.
- Grant: This works & would allow us to make progress, but think it's a strong candidate for a new binding.
- Stefano: The awkwardness is a good reason to simplify.
- Grant: It's an enumeration. You have a range of interrupts & you want to take the group & send them somewhere else.
- Rob: GPIO maps address this already. Has pass through mask or property so you can pass through the GPIO flags. You could do that with interrupt # & flags. That would greatly shorten the map. It's generic now in DT spec: host 0.2 edition of spec. There haven't been lots of changes, so should be easy to find.
- Stefano: Take a look at DT spec for GPIO maps
- Grant: This could be somewhere that needs new binding. Moving into a new node does make sense b/c the device still has only 1 output line. Probably only need 1 cell to describe the input to the distributor node b/c there is no state associated (don't care if level or edge). Think only need the fields if it's a real interrupt controller - what you have here is probably just wire going to multiple GICs.
- Refer to "interrupt-multiplex" in the example
- There will be more things that we'll need, but this is the minimal set to be able to describe interrupt & address map. Now we have technique to do both.
- Loic: How interrupts are defined in the different peripherals which could be accessed by both CPUs?
- Stefano: all the interrupt maps under imux 0x14 to A72 GIC & 0x14 to R5 GIC
- Grant: Not sure what will happen with current software when it finds same interrupt mapped to 2 different places. Not clear spec-wise what should happen.
- Stefano: Read spec. Don't think this is covered. This is reason to come up with different binding, so we can clarify what should happen. Don't have a strong opinion - happy to follow your advice.
- Under interrupt-multiplex have interrupt-parent: There are 2 (gic_a72 & gic_r5). Not sure if that's OK.
- Grant: Don't think it's required
- Rob: Interrupt controller should not be there either. Looks for either interrupt controller or interrupt map.
- Stefano: Now only uncertainty remains: What if put 2 times under interrupt map
- Grant: Does spec require numerical order?
- Stefano: No, could have put 0x14 A72, 0x14 R5. Not sure if could get rid of 1st & 2nd zero field.
- Grant: You should be able to just have 1 cell & have it be a number…. Or, maybe need flags field if have mix & match of level & edge interrupts going through
- Stefano: Would like to come up with best possible binding. We have option to carry flag field. No simple way to show interrupt being duplicated. Still need the 3 cells on the right, which are duplicate of device side. Let's take this offline.
- Loic: Current proposal lists all interrupts in one unique map (1st you have all A72 then R5) and so SW will have to parse all map to find configuration associated to physical interrupt controller. Would it be possible to have a map table per interrupt parent and to select the right table according to selected parent? A bit like GPIO for which we have several states possible? Can we have interrupt map 0 for A72 & map 1 for R5 and then say 1 will use according to parent.
- Rob: Then you could extend interrupt-parent to have more than 1 phandle. We probably have to support both the case where the interrupt #s are all the same & case where different.
- Loic: Can consider virtual lines & physical lines and do the translations to simplify
- Rob: What do you mean by virtual line? Make up some #s for DT?
- Loic: Yes
- Stefano: Currently, the # on the left is a fake #. That 0x14 on the left is matching the device. We care about what's going to the parent - the # on the right. Not saying I like this, but describing how it works. It's essentially a virtual interrupt b/c doesn't exist anywhere.
- Rob: OK, if there's 2 number spaces then need a 3rd to be common.
- Stefano: Will look at GPIO controller extensions & write in detail & send to list so we can discuss. Meanwhile, if you can think of simple solution to this, let Stefano know & he will try to add to the example. In next couple weeks.
- Stefano: There is duplication that's not ideal. It gets us going, and might work, but there is room for improvement.
- Stefano: There is more in the example, but just covered the highlights relevant to System DT for discussion today
- Rob: What about memory nodes? What is the R5 using for memory?
- Stefano:
- Haven't ported all the OpenAMP domain things. In our system, the R5 sees all the RAM the same as the A72.
- For the carve out, those are configurable & done w/ reserved memory & openAMP stuff.
- The only interesting bit for memory is list of TCM regions, which showed as example in previous discussions. It's only a few hundred KB & shows up at 2 addresses on Cortex-R & 1 on the Cortex-A. Don't have it here b/c not in Versal DT yet. Would be under amba & under address-map for cluster, or amba_rpu will make it appear in 2 locations for R5. See TCM comment in example under cpus_r5.
- Stefano:
- Stefano: Worked with Grant to get system DT to work in default case
Tomas, Nathalie, Stefano, Arnd, Maarten, Dan (Mentor), Grant, Ed, Vincent, Matthew, Kumar, Azzedine, Jeremy, Dan (Wind River), Loic, Erwan, Steve, Bruce, Mark, Joakim, Achin, Sudipto, Anders, Manju, Saravana, Arvind, Krzystof, Clement, Mark (Xilinx), Etsam, Waqas, Vivek, Bjorn, Rob, Trilokkumar, Etienne, Victor
Dive into the details of what we have done so far, get feedback, who is interested in engaging, ask questions, identify open issues to work on
- 2 parts
- What changes needed to spec
- Configuration
- Stefano: showed proposed changes to spec using example code
- Arnd: What about interrupts?
- Stefano: idea is to use interrupt extended property
- e.g. R5 cluster will have different GIC from A53 cluster
- Lopper will be able to chop out interrupt hierarchy that's not relevant to you CPU cluster
- Stefano: idea is to use interrupt extended property
- On use of term "ranges"
- Rob: We should not call it ranges b/c it's not ranges. Something map, but map has a defined meaning as well.
- Grant: If you put the full system tree to OS, will it still make sense? Do you need lopper? Want simplest case to work with System DT as given. Want additive & not transformative. Not opposed to changes to Linux. Would like there to be very high bar for proposal that breaks Linux (e.g. go to 2 step tool). Only if it's the only way to do it.
- Rob: We should minimize the processing that takes place to create a bootable DT
- Bjorn brought up that UART might require processing
- Stefano: Need 1 more hook for interrupts. Don't think 0 changes to Linux is possible.
- Arnd: Will depend on platform.
- Kumar: Why not be explicit, like interrupt extended property. We get in trouble with defaults. Being explicit removes question of wanting root cluster a certain way. "cpu" cluster has not kept up & is antiquated. To maintain compatibility, we should be more explicit in map property & then you don't have to worry about which cluster. Map applies to CPUs and which it applies to.
- Ed: The current system is broken for someone trying to use non-Linux OS on a heterogeneous system. 1/3 time of OpenAMP development is trying to get Linux DT to match what you're doing. There's a lot of breakage on the other side of the fence.
- Grant: Let's take it offline. This is awfully close. If a53 was just named cpus and in common case where there isn't a CPU map, it works. Let's go through some scenarios when it gets concrete. We know it's an issue.
- Tomas: System DT is not for everyone. You can skip it if you don't need it, and continue to use include files. Ideal if you could make it backward compatible.
- Grant: Would like to see it as core part of DT tooling. Don't want another tool off to the side.
- Tomas: Agree
- Azzedine: Does it mean we have action to define what mapping needs to be?
- Stefano: Yes
- Software configurations
- Chosen node -> SW configuration
- What kernel runs where
- Which resources supposed to & not supposed to use. No HW limitation in usage of the resources, it's SW
- Kumar & Stefano: on where to draw the line for what goes in "chosen"
- Stefano has Secure/Non-secure and R5 lockstep covered under chosen. Lockstep is strange b/c affects CPU configuration.
- Kumar: Don't think you should put info about how the HW works in chosen. Chosen is for configuration. HW map in secure should be natural DT.
- Stefano: Some things are in grey area - this is one. If we see System DT as static that comes from manufacturer in your box with your kit. The board in the box can be one or the other. It's the customer that will decide if lockstep or not.
- Kumar: Then every config option now has to move here. Don't think that's feasible. Either you have references to everything or duplicate everything. You'll have 1001 properties you'll have to configure this way.
- Kumar: Whether it's a CPU or a UART, it's just a sub part of a system and has properties & can have defaults that cause behavior to occur.
- Stefano: Will take this as a note to think more on it.
- Tomas: Somewhere all this info needs to be specified. Shouldn't be device mfrer, should be customer. We started with a YAML file & lopper can handle. We decided to continue using DT syntax b/c easier for the tool. We need to separate personas doing different things. Don't want to make someone change in many places for 1 thing. Should we have a separate format? That's a separate discussion.
- Kumar: There is a DT initiative around configuration. As we have schema info for the bindings, how to specify if property needs to be configured. Should user specify? Should it be expert mode? What is more convenient to user?
- To consider: OS or firmware as consumer?
- Chosen node -> SW configuration
- Tomas: Have added some real use cases around OpenAMP & realize that what, for e.g. Xilinx, ST, TI, need are different
- Should Linux be able to handle system DT without lopper? Is it viable? Do we need to change Linux?
- How to handle interrupt controller?
- Ranges map - should it be called something different
- Secure: Where to put the info - hw vs. configuration. Don't want to duplicate notes. Want SoC vendor to be able to describe the HW & say what you can do. What should input be? DT format, YAML
- Rob: Suggestion for process
- Better if we do implementation details on email review in device tree spec list
- In the call, stick to more high-level things like what do we want to solve
- Ranges
- Interrupts
- How do we assign devices
- Azzedine: Pick 1 topic at a time to focus on in the follow up meetings.
- Tomas: There are lots of ppl who are interested -> should recommend they join the device tree spec mailing list
- Stefano: Virtualization - memory property - address range
- Azzedine & Stefano to discuss offline
- Logistics: When to meet at Linaro Connect? When is a good time to have regular meetings, how often, other people to involve, ... - Tomas/Nathalie
- Quick recap of the problems that System-Device Trees are solving - Tomas
- Proposed additions to specification to describe multiple CPU clusters - Stefano
- Proposed conventions to allocate resources to domains - Stefano
- Update on Lopper tool - Bruce
Arnd Bergmann, Arvind (Mentor), Benjamin Gaignard (ST), Bill Fletcher - Linaro, Bill Mills (TI), Bruce Ashfield, Clément Léger (Kalray), Dan Driscoll, Danut Milea, Ed Mooring, Etsam Anjum (MGC), felix, Iliad, Jan Kiszka, jeremy gilbert (Qualcomm), Joakim Bech, Loic Pallardy (ST), Maarten Koning, Manju kumar, Nathalie Chan King Choy, Stefano Stabellini, Steve McIntyre, Suman Anna (TI), Tomas Evensen, Tony McDowell (Xilinx), Trilok Soni
- Tomas: Challenge with System DT is that it's cross-functional, so no 1 single mailing list to discuss
- Want to start with some live discussions
- Will formalize proposal & take to mailing lists
- Anyone opposed to recording: No. Recording starts here.
- Intros by attendees
- Tomas: Quick intro on System DT
- Background:
- SoCs very heterogeneous w/ multiple CPU clusters
- System SW needs info about the system (registers in memory map, topologies, DDR allocation, etc.)
- Different personas involved
- HW architect
- System architect
- Info for each operating system
- Config info comes in at different times: Sometimes the config info is compiled in, sometimes loaded at boot, etc.
- Not just multiple CPUs
- other accelerators may need memory
- Execution levels
- Trustzone
- Today, how this is specified is ad hoc. Want 1 true source & tooling that can get the info to the right place.
- A lot of RTOS vendors started to do work with Device Trees, so a natural source
- Verification is also a really important part of it
- At Xilinx, had an XML file & tooling to give the info to Linux, baremetal, free RTOS
- Not standardized
- Need an industry standard. Device Tree is almost there.
- DT looks at 1 address space, what Linux sees or what Xen sees.
- Need to expand DT -> System DT
- It should do 2 things:
- Describe HW for all the different masters, or CPU clusters
- Need to add some things into devicetree.org spec & tooling
- Describe AMP config info: what goes where & domain
- Suggesting "chosen" section
- Want to be compatible with hypervisor. (Stefano a maintainer with Xen, so makes sense to align)
- Describe HW for all the different masters, or CPU clusters
- It should do 2 things:
- Lopper creates traditional DT based on System DT
- So, we don't have change the DT that Linux or U-Boot sees
- Maybe longer term, Linux may want to know about System DT
- System DT data flow (RTOS & Linux) diagram showing different personas & sources of info
- Can add new nodes, add attributes into node, suppress things
- System architect info can be added as snippet
- Want to change 1 part of system without editing everyone else's files, but could still be merged together later
- Showing some possible flows for RTOS/baremetal & Linux
- Example: System DT + domain info from system architect -> Linux DT using Lopper
- Flow diagram for Xen & QEMU
- Intent to get it to work with System DT in future
- What is shown with QEMU is Xilinx QEMU. Upstream does differently. Trying to get Xilinx method upstream.
- Domains & resource groups
- Is what you are sharing intended to be shared?
- Background:
- Questions for Tomas about vision?:
- No questions
- Tomas: We found a lot of ppl have solved similar problems, which makes it interesting
- Mentor has some tooling & syntax to specify allocations. Ahead of where we are in terms of the problems encountered. Looking forward to your feedback.
- Wind River using DT for VxWorks in different way
- How do you distinguish between in TZ & outside TZ?
- Really want this to be a universal solution as much as possible
- Face to Face at Linaro Connect
- Will have possibility for people to call in
- Wed afternoon or Thur morning
- Steve: Suggest Wed 3:30pm PDT for SystemDT but OK to move
- Who would be calling in from other time zones? Will have Europe
- Morning is boot session from LEDGE, maybe could swap? Wed morning is kernel & ppl need to be there.
- 9am-10:30am PDT Thur morning looks most promising. 16:00 UTC
- Steve will try to make these show up on public schedule
- Perhaps may have ~30 ppl
- Stefano: Representative example (not complete) of System DT that would be input to Lopper
- Output of Lopper could be used by Linux, or RTOS
- 1st set of changes: How to represent CPU clusters
- DT mandates only 1 CPU notes called "cpus"
- We want to represent A53 cluster & R5 cluster, which have different architectures
- Want to introduce concept of CPU cluster -> small change to spec, but deep impact to DT
- Keep in mind: Different CPU clusters can have different memory maps
- Example on Xilinx board where A53 memory map is different from R5 memory map: e.g. tcm_bus region
- Available at 2 addresses for R5 cluster (low address and high address)
- Purpose is to show different memory map for 2 different CPU clusters
- ranges-map is showing AMBA is accessible 1:1 & TCM bus mapping in A53, in R5 have additional entry for TCM
- These are changes that we need for System DT that can represent multiple heterogeneous CPUs
- Nothing about configuration shown here
- Domain section under "chosen": How to explain configuration
- Could be changed in SW or firmware
- What is configured to run where
- What privilege level on CPUs
- What shared memory we will have between different domains (area where 1 set of SW runs, e.g. R5 domain vs. A53 domain)
- We have a memory property that shows address & size for RAM region accessible from this domain
- CPU attribute is interesting b/c tells us where this domain is running (e.g. R5 cluster & CPU mask shows which CPU used to run this domain)
- Architecture-specific # can represent an architecture-specific config of the CPU
- e.g. ARM R5 can be lockstep or not lockstep and secure mode or normal mode
- Other key part is the "access" property
- Expressing in list of links what is accessible from this domain
- On Xilinx & other vendor boards, there is possibility to enforce isolation
- This expresses if this OpenAMP domain can access
- Example: Showing normal memory, TCM bus, reserved memory region for communicating with A53, IPI is Xilinx HW block for communication between A53 & R5 cluster
- Flag: provide extra info
- e.g. mailbox: Which channel, TX or RX
- Each vendor could express vendor-specific meaning for flag to say how mapping should be done
- Could then be translated into remoteproc configuration as used by Linux. Conceptually maps to the info we saw before. Could be generated by Lopper from System DT.
- Stefano has been getting inputs from different groups & starting to reach out wider.