DSL evaluation for arduino - please take a look

[dzw1999]

Hello, greetings from the software engineering group from Beihang University, China. We have been working on the evaluation of Domain Specific Modeling Languages (DSMLs ), the main purpose is to find out the flaws of the design of the DSML. To test our evaluation framework, we evaluated your software design, mainly based on your ecore and odesign files, and found out that even though the overall quality of your software design is very good, there still exists some minor problems. The details are as follows:

rule-based evaluation

Firstly, we did some rule-based evaluation, mainly focusing on the graphic part.

Evaluation of Semiotic Clarity

We've checked your odesign file according to Goodman's theory of symbols[Goodman N. Languages of Art: An Approach to a Theory of Symbols[M]. Hackett publishing, Indianapolis, 1976.], and there are some minor problems that can decrease the semiotic clarity of your design as follows:

1. overloading

   There are some graphic symbols in your design that is assigned to more than one element.

   • arduino\plugins\fr.obeo.dsl.arduino.design\icons\Sensor.png is assigned to arduino.ParameterCall/arduino.ParameterDefinition
   • arduino\plugins\fr.obeo.dsl.arduino.design\images\delay.svg is assigned to arduino.ParameterCall/arduino.Delay/arduino.ParameterDefinition
   • arduino\plugins\fr.obeo.dsl.arduino.design\icons\Status.png is assigned to arduino.ParameterCall/arduino.ParameterDefinition

2. Excess

   There are some graphic symbols in your design that does not represent any of the elements of your language.

   • arduino\plugins\fr.obeo.dsl.arduino.design\icons\LowerOrEqual.svg
   • arduino\plugins\fr.obeo.dsl.arduino.design\images\platform.svg
   • arduino\plugins\fr.obeo.dsl.arduino.design\images\dfrobot\dfrduino_uno.jpg
   • arduino\plugins\fr.obeo.dsl.arduino.design\images\existing_sketch.png
   • arduino\plugins\fr.obeo.dsl.arduino.design\images\upload.svg

3. Deficit

   There are some elements in your design that are not represented by any graphic symbols.

   Certainly, we know that some elements do not need to be assigned symbols to, but we believe it is best to check whether these elements really do not need symbols or they were just overlooked.

   • DigitalPin
   • Pin
   • AnalogPin
   • Utilities
   • IO
   • Time

Evaluation of Symbol Similarity

We believe that overly similar symbols are not conducive to users' understanding and use of language. Therefore, we use image features such as SSIM to detect the similarity between the symbols you use. Below are some pairs of symbols that may be too similar:

1. arduino\plugins\fr.obeo.dsl.arduino.design\icons\UseFunction.png & arduino\plugins\fr.obeo.dsl.arduino.design\icons\Function.png
   

LLMS-based evaluation

Secondly, we did some LLMS-based evaluation, mainly focusing on the model part. Generally speaking, we try to make LLM act as a domain expert to provide multidimensional evaluations of your language design.

Model Completeness

We give your ecore design to LLM and ask the LLM to guess which domain your language is designed for. And then we ask LLM to add possible missing elements to your language (perhaps not taken into account in the first version of the design, but can be considered for inclusion in future versions). The results are as follows:

These metamodels seem to belong to the field of embedded system programming or microcontroller programming, especially those related to the Arduino programming environment. This is because elements such as "DigitalPin", "AnalogPin", and "Module" are mentioned, which are common concepts in the field of microcontrollers and embedded systems. In addition, "Sketch" is a term used in the Arduino development environment to refer to program code.

In addition to the already listed metamodels, the following elements and their relationships can also be supplemented to help improve the description of this field:

Element:

CommunicationProtocol:
Name;
Version;

WirelessModule:
Frequency;
Range;
Protocol;

DisplayModule:
Resolution;
ColorDepth;

Motor:
Speed;
Type;

SensorType:
Measurement;
Range;

Actor:
Type;

Event:
Trigger;
Action;

Interrupt:
Source;
Priority;

Relationship:

Module (n)<- supports ->CommunicationProtocol (n)
WirelessModule (1)<- communicationProtocol ->CommunicationProtocol (1)
Hardware (1)<- WirelessModules ->WirelessModule (n)
Hardware (1)<- displayModules ->DisplayModule (n)
Hardware (1)<- motors ->Motor (n)
Sensor (1)<- sensorType ->SensorType (1)
Hardware (1)<sensors>Sensor (n)
Hardware (1)<- actors ->Actor (n)
Instruction (1)<- event ->Event (1)
Sketch (1)<- interrupts ->Interrupt (n)
Interrupt (1)<- sensorTrigger ->Sensor (1)

These supplements can more comprehensively cover concepts in the field of embedded systems or microcontroller programming, from communication protocols to various modules (such as wireless modules, display modules, etc.), to event or interrupt based programming patterns, all of which are indispensable elements in this field.

We are not sure if these issues actually constitute a problem, the decision to fix them or not is still up to your team, looing forward to your respond, thanks!