I originally created this as a short to-do list of study topics for becoming a software engineer, I am creating this list to share my 3 year list of topics that I studied on the side during the curriculum of a Computer Engineering and Informatics degree. This is a list of short and medium length study topics to obtain knowledge regarding autonomous rotorcraft. The items listed here will give you enough knowledge to be able to understand how they work, their limitations and effort required to make them fly. Happy studying!
Format of this is hugely inspired by jwasham/coding-interview-university
Contributions are welcome, so please open a PR if you can improve this list or something is missing!
This is my big study plan for going from simple programmer dude to software engineer who can understand how autonomous drones work.
There are references and video of various popular open source flight controller firmware and guidance projects in order to give the reader a sense of realism and real-sense and real-world value of operations, as well as act as get-to-know-the-market info.
This is meant for software engineers or people who already know how to code and also got basic knowledge of computer science topics including math (probablility, calculus, linear algebra). If you have many years of experience this can be easier to read through, but this is not guaranteed.
Disclaimer: I am not employed nor have I got any work experience on commerial or research rotorcraft.
When I started this project, I didn't know how a drone could see, how the math behind the control algorithm worked, what the limitations are or anything like that. I started skimming through any resources I could after being inspired by the newly released Skydio r1 and it's CEO's numerous guest lectures at youtube. I slowly started building enough sensors to be able to grasp every sub-component of such flying systems. I have not learned everything yet, but the list is big enough to share and will always be evolving with more stuff added.
It's a long plan. It may take you months. If you are familiar with a lot of this already it will take you a lot less time.
Everything below is an outline, and you should tackle the items in order from top to bottom.
I'm using Github's special markdown flavor, including tasks lists to check progress.
Create a new branch so you can check items like this, just put an x in the brackets: [x]
Fork a branch and follow the commands below
git checkout -b progress
git remote add zarkopafilis https://github.com/zarkopafilis/awesome-dronecraft
git fetch --all
Mark all boxes with X after you completed your changes
git add .
git commit -m "Marked x"
git rebase zarkopafilis/master
git push --force
More about Github-flavored markdown
- Successful software engineers are smart, but many have an insecurity that they aren't smart enough.
- The myth of the Genius Programmer
- It's Dangerous to Go Alone: Battling the Invisible Monsters in Tech
- Believe you can change
- Think you're not smart enough to work at Google? Well, think again
You can't use only 1 language you are comfortable in to do the coding of the drones. You'd have to rely on many tools. Here are some for example.
- C++
- C
- Python
- Matlab (Yes, I know)
You may also need to learn a bit of other tools in the proccess. This is NOT something that you should feel worried about. With studying consistency there is nothing you can't achieve!
- Quan, Quan - Introduction to Multicopter Design and Control       A very good all-around book. It provides a more than quick-overview on all the different topics regarding quadcopters, from frame design, to electronics and control, diving deep enough on technical and practical design practises and considerations. It provides intuition behind every subject matter and does not containt ambiguities. The mathematics may seem too hard to deal with for some people, but that's not a thing to worry about. A person without mechanical or electrical engineering background can easily tackle this with some calculus and basic geometry knowledge.
This list grew over many months, and yes, it kind of got out of hand.
Here are some mistakes I made so you'll have a better experience.
Take breaks, write down possible things you need to re-review and watch them later or in a different way. This is a lot to take in and breaks are mandatory.
Some concepts can take much longer to understand fully. So if you don't understand it immediately do not worry. I suggest re-studying some of the parts that seemed hard to you after doing one pass on this study list.
There are a lot of distractions that can take up valuable time. Focus and concentration are hard.
- Don't know yet
Learn to code. This is only required for you to understand how the different algorithms and techniques that will be presented are actually implemented. You need this knowledge to be able to understand the source code of the popular data structure implementations, open source flight controllers and more. Basic data structure and algorithmic complexity should be included.
- C, C++, Python, Anything
- Information available on the internet is widely available and you can find lot's of stuff by googling.
- I could recommend to start with Python and work your way down to C and C++.
- You can learn to code in parallel with this study plan, but things will be a lot harder and will take more time.
- A complete computer science study plan to become a software engineer
The next part is needed to be able to understand the math behind the stuff we are going to use. It's not at all harder compared to other fields of study. Hang on and in the end, you'll be surprised by how easy it is to understand everything. This is not the only path to learn these, it's just what I would take. (Yes, I like university lectures and whitepapers.)
- Basic linear algebra, calculus, probablility and elementary physics
- Single Variable Calculus - MIT OCW
- Multivariable Calculus - MIT OCW
- Gilbert Strang's Linear Algebra - MIT OCW there is really no better way to learn this in the world
- Introduction to Probability and Statistics- MIT OCW
- Digital Signal Proccessing - Rensselaer Polytechnic Institute
Optional
-
These can get expensive, but here is a list of the popular ones. Most are cheap or free.
-
This is the basis of rotorcraft that is going to get covered first. These few resources will make you understand what a drone needs to fly bad, good, with the help of extra autonomy engines or with the help of a pilot.
-
After this part, the corks and screws of each subsystem is going to be investigated thoroughly.
At this point, you might wonder: This is only for four rotors. Don't worry, the extra ones are only used to have resilience. In the future this is going to be populated with more types of rotorcraft like submarines, VTOL drones and wings.
-
Now that you have got a rough understanding of how you are going to make things fly, it's time to take a better look into the mathematical concepts behind the control of systems.
-
In order to get the control theory to work, we have to obtain a best-effort state estimate by observing our rotorcraft system.
-
Kalman Filters
-
Map Projections
-
Inertia Measurement Units
-
Cameras and Optical Sensors
- How does a camera work
- Rolling and Global Shutter
- Optical Flow
- Depth Perception
- PX4: Gimbal Control (site)
- MAVLink: Gimbal Protocol (site)
-
Corrections and Calibration
-
Communication Protocols
-
The sensor subsystem is complete. We can now procceed to construct a map of the environment as well as figure out where our UAV is located in the world.
-
Octomap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees (site)
-
Autonomous Driving: Localization and Deep Learning Powered Mapping
-
After proper localization, we are ready to path or mission plan, to make the UAV complete a predefined course of trajectory along with actions
-
3DVFH+: Real-Time Three-Dimensional Obstacle Avoidance Using an Octomap (paper)
- How brushless motors work and how to control them
- Advantages and Disadvantages of Brushed and Brushless Motors
- How to choose the right brushless motor for your drone
- Slow retracting landing gear for drones
- Drone with Dual Robot Arms
-
For your own interest. Take a look into the source and issue tracker, maybe join the weekly dev calls.
-
Autopilots:
-
SDKs:
-
Ground Control Stations:
-
Simulators:
-
Middleware:
-
Protocols:
-
Design Tools:
-
Flight Evaluation - Rule and Constraint based airframe design and evaluation
-
For your own interest as well.
- Betaflight: Software Architecture Diagrams
- Betaflight: Unified Board Targets
- Pixhawk: Architectural Overview
- ArduPilot: Flight Modes
- Pixhawk: Flight Review
-
In order to do that, you have to find a set of required hardware that is compatible, burn and configure your desired flight controller firmware.
-
Minimum Hardware
- Frame. For racing/fully manual control, typically carbon fiber, but you can even 3d print one or carve one out of wood. Take 2 dimensions in mind: flight controller stack size and propeller size. Most common propellers would be about 5".
- Flight Controller. You know enough to choose one. I'd use betaflight.
- ESCs. You can go with a 4 x 1 ESC or All in one board combo. Each has got advantages and disadvantages, search about it. But first, check your battery. It should be able to handle the burst current.
- PDB. You need a power distribution board. These are relatively small and uncomplicated. You can find PDB+ESC+FC all in one combos that include battery step down and all that. But first, check your battery.
- Battery. LiPo. At the time of this writing you can go from 4s to 6s. Price depends on capacity, discharge rating and cell count.
- Brushless Motors. You know, to make the thing move.
- RC Transceiver. Should be compatible with your RC Controller and it's protocol should be compatible with your FC board. Typically 2.4GHz.
- RC Controller. Flysky, Taranis, there are too many to list.
- After configuring frameware you should be able to fly this, but there are several stuff missing.
-
Optional Hardware
- Camera. Not the kind you are used to. This one sends out (analog) PAL/VTSC video and should support OSD
- VTX. Stands for video transmitter. This is used to broadcast the video to typically 5.8GHz. Research how much mW you need.
- FPV Goggles. To view the video. Should bear compatible antenna with the VTX. Diversity modules which dynamically choose the best signal source from an omnidirectional and a directional antenna are better.
- Aftermarket Antennas. For beter reception.
- Beeper. Beeps if you lose it.
- Battery Balance Charger. You can't use discharged batteries.
- Battery Balance Charger Power Supply. If your charger does not come with one. (Most IMAX B6s do not, for example)
- 3D Printed GoPro Session Case
- Insulation Tape
- Zipties
- LEDs for the bottom of the craft
- Velcro Battery Straps
-
Software and Further Configuration
- Blackbox Logging. Important information regarding your flight. Can help you out for post-flight analysis.
- RC Channel Mapping
- Make sure that ESCs spin in the correct directions. You can change this in the ESC software or flight controller software.
- Pair RC Transceiver with your RC Controller.
- PID and Rates Tuning
- Keep a configuration backup along with the exact firmware number so you don't forget.
Experienced racing pilots recommend that you stard by flying 'acro'/'freestyle' mode straight from the start. Do not do that with real drone hardware, the probability of crashing within 2 seconds is almost 100%. Take your time on a simulator first. There are many ones that you can find on the most popular gaming platforms like Steam. For example: Liftoff, DRL Sim, etc...
Flying 'angle'/'aided' mode is much easier and the same to flying your typical DJI/Parrot/etc drone around. You can fly this 'line of sight' even if you dont have goggles.
-
Building Fully Autonomous Rotorcraft is much harder and more resource intensive than building a computer-aided one. You need to do better calibration because there is no pilot directly flying it-only supervising it.
-
Basic Hardware. Basic in the sense that you need at least those for this to fly autonomously. You could maybe leave the GPS out of that, but you need at least optical flow for exact stabilization.
- Flight Controller. Prefer a pre-tuned pre-built one that's resilient and robust enough. The Pixhawk 4 is a very good option, with dual IMUs, a ton of sensors, standarized connectors and a ton of connectivity options.
- GPS
- and/or Bottom Mounted Camera(s). Example of a drone that has got cameras on the bottom but no GPS, is the one from the matlab tech talks.
- Frame
- Propellers
- Brushless Motors
- Power Distribution Board
- LiPo Battery
- Connectivity
- Some way to send mission control data and receive information
- RC Controller to pilot-override the trajectory
-
Optional Hardware
- Basic Obstacle Avoidance
- Front Mounted Camera (or RGB-D or long range ultrasonic for slow moving drones)
- Basic Co-Processor if the flight controller firmware can't handle obstacle avoidance on it's own
- Auto Land and Hand Land
- Bottom mounted ultrasonic sensor
- and/or bottom mounted camera(s) -- two cameras needed for depth in order to be more reliable than optical flow
- Payload Carry
- Some kind of magnet or servo to drop payloads
- Advanced Camera Ops
- Rotating Gimbal. In order not to couple the field of view along with the movement of the drone.
- RGB-D Camera (for better SLAM)
- LiDAR (SLAM too)
- Retracting Landing Gear
- Extra Connectivity
- WiFi
- LoRa
- E-3G-4G-5G
- Bluetooth
- Basic Obstacle Avoidance
-
Useful Co-Proccessor Compute Boards
Usually, you start of from the 'job' requirements ex. Mapping, Crop Spraying, Inspection and then decide what hardware is capable of doing this job on a cost-benefit scenario. Configure everything and you should be ready to go.
Congratulations!
Keep learning.
You're never really done. But still, good job :).
-
How to Make an Infared Night Vision Camera From a Regular Digital Camera
-
The Reconfigurable Aerial Robotic Chain: Modeling and Control
- Wings
- VTOL
- Underwater
-
UAV Swarms
-
Automatic Resupply and Mission Resum
-
Deep Learning
Now that you have made it so far, here are the last resources I am able to point you to. From this point on you should have a good understanding of how drones work and how to have a digital perception of the environment. The last parts are exactly a few missing pieces and how to pull everything together:
- Efficient and Accurate Tracking Objects Through Video
- Kernelized Correlation Filters. This is the efficient approach #1. A very simple training step with a very simple detection and on-line update step.
- End-to-end representation learning for Correlation Filter based tracking. This is the step #2, sacrificing some computational power and complexity in order to get a more robust and transformation-occlusion invariant tracking
- Lightweight Deep Neural Network Computations for Mobile and Embedded Platforms
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. There is also a V2 version of MobileNets.
- Intel OpenVINO. Optimizing deep learning models for the edge. nVidia and other companies provide their own toolkits as well, but they all work on a similar pruning-and-reducing neurons while maintaining a high enough performance metrics.
- Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments. AKA the Skydio paper. The authors of this paper have published a lot of work on the subject and will probably keep doing them. I suggest that you look the up on google scholar.
Lastly, I hope that enjoyed reading and learning from this simple text file as much as I enjoyed researching and writing about it.