-
Notifications
You must be signed in to change notification settings - Fork 0
/
webcam-midi.html
242 lines (218 loc) · 15.1 KB
/
webcam-midi.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>David Spielman</title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/WebsiteLogo2_thicker.png" rel="icon">
<link href="assets/img/apple-touch-icon.png" rel="apple-touch-icon">
<!-- Google Fonts -->
<link href="https://fonts.googleapis.com/css?family=Poppins:300,300i,400,400i,500,500i,600,600i,700,700i|Playfair+Display:400,400i,500,500i,600,600i,700,700i,900,900i" rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<!-- =======================================================
* Template Name: Folio - v4.3.0
* Template URL: https://bootstrapmade.com/folio-bootstrap-portfolio-template/
* Author: BootstrapMade.com
* License: https://bootstrapmade.com/license/
======================================================== -->
</head>
<body>
<!-- ======= Header ======= -->
<header id="header" class="fixed-top ">
<div class="container d-flex align-items-center justify-content-between">
<!--<a href="index.html" class="logo"><img src="assets/img/logo.png" alt="" class="img-fluid"></a>-->
<!-- Uncomment below if you prefer to use an text logo -->
<h1 class="logo"><a href="index.html">David Spielman</a></h1>
<nav id="navbar" class="navbar">
<ul>
<li><a class="nav-link scrollto active" href="index.html">Home</a></li>
<!--<li><a class="nav-link scrollto" href="#about">About</a></li>-->
<li><a class="nav-link scrollto" href="index.html#portfolio">Portfolio</a></li>
<li><a class="nav-link scrollto" href="David Spielman Resume.pdf">Resume</a></li>
<!--<li class="dropdown"><a href="#"><span>Drop Down</span> <i class="bi bi-chevron-down"></i></a>
<ul>
<li><a href="#">Drop Down 1</a></li>
<li class="dropdown"><a href="#"><span>Deep Drop Down</span> <i class="bi bi-chevron-right"></i></a>
<ul>
<li><a href="#">Deep Drop Down 1</a></li>
<li><a href="#">Deep Drop Down 2</a></li>
<li><a href="#">Deep Drop Down 3</a></li>
<li><a href="#">Deep Drop Down 4</a></li>
<li><a href="#">Deep Drop Down 5</a></li>
</ul>
</li>
<li><a href="#">Drop Down 2</a></li>
<li><a href="#">Drop Down 3</a></li>
<li><a href="#">Drop Down 4</a></li>
</ul>
</li> -->
<li><a class="nav-link scrollto" href="index.html#contact">Contact</a></li>
</ul>
<i class="bi bi-list mobile-nav-toggle"></i>
</nav> <!-- .navbar -->
<div class="header-social-links">
<a href="https://www.linkedin.com/in/david-spielman/"><i class="bi bi-linkedin"></i></a></li>
<a href="https://github.com/DavidSpielman"><i class="bi bi-github"></i></a></li>
<a href="mailto: [email protected]"><i class="bi bi-envelope"></i></a></li>
</div>
</div>
</header><!-- End Header -->
<main id="main">
<!-- ======= Blog Single ======= -->
<div class="main-content paddsection">
<div class="container">
<div class="row justify-content-center">
<div class="col-md-8 col-md-offset-2">
<div class="row">
<div class="container-main single-main">
<div class="col-md-12">
<div class="block-main mb-30">
<img src="assets/img/portfolio/Webcam_Midi_Controller/webcam-midi-project-icon.jpg" class="img-fluid" alt="">
<div class="content-main single-post padDiv">
<div class="journal-txt">
<h4><a href="#">Webcam MIDI Controller</a></h4>
</div>
<div class="post-meta">
<ul class="list-unstyled mb-0">
<!--<li class="author">by:<a href="#">medsign</a></li>-->
<li class="date">date: July 2021 - Present</li>
<!--<li class="commont"><i class="ion-ios-heart-outline"></i><a href="#">3 Comments</a></li>-->
</ul>
</div>
<style>
p {
color: black;
}
</style>
<p class="mb-30">In my free time I enjoy composing music. The inspiration for this project came about as I was brainstrorming interersting ways to make and play music. When I was younger, I remember enjoying outdoor interactive music installations at children's science museums. I wanted to recreate the satisfaction of striking a chime or PVC pipe with a paddle at home, so I began working on this MIDI controller.</p>
<p class="mb-30">This project incorporates OpenCV, ROS, a Teensy 4.0 microcontroller, a Logitech Pro 9000 USB webcam, the Reaper DAW (Digital Audio Workstation), and a table tennis paddle. OpenCV tracks the position of the table tennis paddle on the live video footage displayed by the webcam, and depending on where the paddle is placed on the screen, a ROS publisher node will publish note data to a subscriber node associated to the Teensy microcontroller. The Teensy microcontroller will then send MIDI data to the Reaper DAW where it will be converted into recordable sound using audio plugins.</p>
<p class="mb-30">To track the position of the table tennis paddle I needed to use a computer vision library. I decided to go with OpenCV since it is open source and is widely used. This was my first exposure to computer vision and OpenCV, so I was excited to start learning through this project.</p>
<p class="mb-30">After about a week and a half of reading through the OpenCV documentation, troubleshooting installation and video capture issues, and consulting many forums and YouTube videos, I learned enough to create the tracker below. It uses OpenCV's built-in CSRT tracker which I found to work best for this application. I placed a Post-It note on the table tennis paddle to improve tracking. I also added a reset function to readjust the selected region of interest (ROI) if the tracker lost the object being tracked. In the GIF below, I selected the ROI to be the table tennis paddle, and then I reset the ROI to track my face.</p>
<img src="assets/img/portfolio/Webcam_Midi_Controller/Tracking Paddle & Reset Test.gif" class="center2" alt="">
<p class="mb-30">Once the tracker was ready, I added an GUI interface on the video displayed by the webcam to help the user select which note interval they wished to play. Based on the location of the center of the ROI (the red dot in the GIF below) OpenCV will publish which interval the user has selected through a ROS publisher node. </p>
<img src="assets/img/portfolio/Webcam_Midi_Controller/Gui_Layout_Test.gif" class="center2" alt="">
<p class="mb-30">The ROS publisher node sends messages through rosserial which is a way for ROS to communicate with Arduino compatible microcontrollers. I chose to use a Teensy 4.0 microcontroller since it can natively send MIDI data through USB, is Arduino IDE compatible, and due to its small size. I have made a ROS subscriber node that works with the Teensy microcontroller, but due to issues with a previous Teensy microcontroller that was defective, I am still working on ways to send MIDI data from the subscriber node to my DAW using a new Teensy microcontroller. (DAW stands for Digital Audio Workstation and it is a software tool used by many musicians to manipulate, record, and process sound to make and release music.) </p>
<p class="mb-30">The next step for this project is to send MIDI data from the Teensy microcontroller to the Reaper DAW. I also plan on adding an IMU to the table tennis paddle to translate how hard the GUI boxes are "hit" to MIDI velocity data. This will allow the user to control the volume of the notes produced by how fast they move the paddle.</p>
<a href="https://github.com/DavidSpielman/Webcam-Midi-Controller"><u>Click here to view the source code for this project</u></a>
</div>
</div>
</div>
<!-- <div class="col-md-12">
<div class="comments text-left padDiv mb-30">
<div class="entry-comments">
<h6 class="mb-30">4 comments</h6>
<ul class="entry-comments-list list-unstyled">
<li>
<div class="entry-comments-item">
<img src="assets/img/avatar.jpg" class="entry-comments-avatar" alt="">
<div class="entry-comments-body">
<span class="entry-comments-author">Sommer Christian</span>
<span><a href="#">fev 14, 2018 at 12:48 pm</a></span>
<p class="mb-10">Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quam justo, ullamcorper tincidunt pellentesque in, condimentum ut enim. Aenean at pharetra diam, quis vulputate urna. </p>
<a class="rep" href="#">Reply</a>
</div>
</div>
<ul class="entry-comments-reply list-unstyled">
<li>
<div class="entry-comments-item">
<img src="assets/img/avatar.jpg" class="entry-comments-avatar" alt="">
<div class="entry-comments-body">
<span class="entry-comments-author">Sara Smith</span>
<span><a href="#">fev 14, 2018 at 12:51 pm</a></span>
<p class="mb-10">Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quam justo, ullamcorper tincidunt pellentesque in, condimentum ut enim. Aenean at pharetra diam, quis vulputate urna.</p>
<a class="rep" href="#">Reply</a>
</div>
</div>
</li>
</ul>
</li>
<li>
<div class="entry-comments-item">
<img src="assets/img/avatar.jpg" class="entry-comments-avatar" alt="">
<div class="entry-comments-body">
<span class="entry-comments-author">Andrew Lupkin</span>
<span><a href="#">fev 14, 2018 at 12:55 pm</a></span>
<p class="mb-10">Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quam justo, ullamcorper tincidunt pellentesque in, condimentum ut enim. Aenean at pharetra diam, quis vulputate urna. </p>
<a class="rep" href="#">Reply</a>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="col-lg-12">
<div class="cmt padDiv">
<form id="comment-form" method="post" action="" role="form">
<div class="row">
<div class="col-lg-6">
<div class="form-group">
<input id="form_name" type="text" name="name" class="form-control" placeholder="Name *" required="required">
</div>
</div>
<div class="col-lg-6">
<div class="form-group">
<input id="form_email" type="email" name="email" class="form-control" placeholder="email *" required="required">
</div>
</div>
<div class="col-md-12">
<div class="form-group">
<input id="form_name" type="text" name="website" class="form-control" placeholder="Website">
</div>
</div>
<div class="col-lg-12">
<div class="form-group">
<textarea id="form_message" name="message" class="form-control" placeholder="Message *" style="height: 200px;" required="required"></textarea>
</div>
</div>
<div class="col-lg-12">
<input type="submit" class="btn btn-defeault btn-send" value="Send message">
</div>
</div>
</form> -->
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div><!-- End Blog Single -->
</main><!-- End #main -->
<!-- ======= Footer ======= -->
<div id="footer" class="text-center">
<div class="container">
<p>© 2021 David Spielman. All rights reserved.</p>
<div class="credits">
<!--
All the links in the footer should remain intact.
You can delete the links only if you purchased the pro version.
Licensing information: https://bootstrapmade.com/license/
Purchase the pro version with working PHP/AJAX contact form: https://bootstrapmade.com/buy/?theme=Folio
-->
Designed by <a href="https://bootstrapmade.com/">BootstrapMade</a>
</div>
</div>
</div><!-- End Footer -->
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/php-email-form/validate.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<script src="assets/vendor/typed.js/typed.min.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
</body>
</html>