Merge pull request #625 from pierotofy/febsprint

February Sprint

README.md

@@ -30,10 +30,9 @@ A free, user-friendly, extendable application and [API](http://docs.webodm.org)
  * [Run it natively](#run-it-natively)
  * [Run it on the cloud (Google Compute, Amazon AWS)](#run-it-on-the-cloud-google-compute-amazon-aws)
 
+![ezgif-1-c81c8daab2e0](https://user-images.githubusercontent.com/1951843/52976882-3db81d80-3399-11e9-8915-ffb00b062aaf.gif)
 
-![Alt text](https://user-images.githubusercontent.com/1951843/28586405-af18e8cc-7141-11e7-9853-a7feca7c9c6b.gif)
-
-![Alt text](/screenshots/pointcloud.png?raw=true "3D Display")
+![ezgif-1-4d8402e295f9](https://user-images.githubusercontent.com/1951843/52976888-43adfe80-3399-11e9-8bc6-1690806131d1.gif)
 
 
 ## Getting Started
@@ -265,25 +264,12 @@ Developer, I'm looking to build an app that displays map results and takes care
 Developer, I'm looking to build an app that will stay behind a firewall and just needs raw results | [NodeODM](https://github.com/OpenDroneMap/NodeODM)
 
 ## Roadmap
-- [X] User Registration / Authentication
-- [X] UI mockup
-- [X] Task Processing
-- [X] 2D Map Display 
-- [X] 3D Model Display
-- [ ] NDVI display
-- [X] Volumetric Measurements
-- [X] Cluster management and setup.
-- [ ] Mission Planner
-- [X] Plugins/Webhooks System
-- [X] API
-- [X] Documentation
-- [ ] Android Mobile App
-- [ ] iOS Mobile App
-- [ ] Processing Nodes Volunteer Network
-- [X] Unit Testing
-- [X] SSL Support
-
-Don't see a feature that you want? [Help us make it happen](/CONTRIBUTING.md). 
+
+We follow a bottom-up approach to decide what new features are added to WebODM. User feedback guides us in the decision making process and we collect such feedback via [improvement requests](https://github.com/OpenDroneMap/WebODM/issues?q=is%3Aopen+is%3Aissue+label%3Aimprovements).
+
+Don't see a feature that you want? [Open a feature request](https://github.com/OpenDroneMap/WebODM/issues) or [help us build it](/CONTRIBUTING.md).
+
+Sometimes we also prioritize work that has received financial backing. If your organization is in the position to financially support the development of a particular feature, [get in touch](https://community.opendronemap.org) and we'll make it happen.
 
 ## Getting Help
 
@@ -304,7 +290,7 @@ There are many ways to contribute back to the project:
  - ⭐️ us on GitHub.
  - Spread the word about WebODM and OpenDroneMap on social media.
  - While we don't accept donations, you can purchase an [installer](https://webodm.org/download#installer) or a [premium support package](https://webodm.org/services#premium-support).
- - Become a contributor (see below to get free swag 🤘)
+ - Become a contributor 🤘
 
 ## Become a Contributor
 
@@ -314,14 +300,10 @@ You don't necessarily need to be a developer to become a contributor. We can use
 
 If you know how to code, we primarily use Python (Django), Javascript (React), HTML and SCSS. See the [Development Quickstart](http://docs.webodm.org/#development-quickstart) and [Contributing](/CONTRIBUTING.md) documents for more information.
 
-To make a contribution, you will need to open a pull request ([here's how](https://github.com/Roshanjossey/first-contributions#fork-this-repository)). To make changes to WebODM, make a clone of the repository and run `./devenv.sh start`.
+To make a contribution, you will need to open a pull request ([here's how](https://github.com/Roshanjossey/first-contributions#fork-this-repository)). To make changes to WebODM, make a clone of the repository and run `./webodm.sh start --dev`.
 
 If you have questions visit us on the [forum](http://community.opendronemap.org/c/webodm) and we'll be happy to help you out with your first contribution.
 
-When your first pull request is accepted, don't forget to fill [this form](https://goo.gl/forms/PZkiPPeNKUHNz0qe2) to get your **free** WebODM T-Shirt 🤘
-
-<img src="https://user-images.githubusercontent.com/1951843/36511023-344f86b2-1733-11e8-8cae-236645db407b.png" alt="T-Shirt" width="50%">
-
 ## Architecture Overview
 
 WebODM is built with scalability and performance in mind. While the default setup places all databases and applications on the same machine, users can separate its components for increased performance (ex. place a Celery worker on a separate machine for running background tasks).

app/api/tasks.py

@@ -2,22 +2,32 @@
 from wsgiref.util import FileWrapper
 
 import mimetypes
+
+import datetime
 from django.core.exceptions import ObjectDoesNotExist, SuspiciousFileOperation, ValidationError
 from django.db import transaction
 from django.http import FileResponse
 from django.http import HttpResponse
 from rest_framework import status, serializers, viewsets, filters, exceptions, permissions, parsers
 from rest_framework.decorators import detail_route
-from rest_framework.permissions import IsAuthenticatedOrReadOnly
+from rest_framework.permissions import AllowAny
 from rest_framework.response import Response
 from rest_framework.views import APIView
 
 from app import models, pending_actions
+from nodeodm import status_codes
 from nodeodm.models import ProcessingNode
 from worker import tasks as worker_tasks
 from .common import get_and_check_project, get_tile_json, path_traversal_check
 
 
+def flatten_files(request_files):
+    # MultiValueDict in, flat array of files out
+    return [file for filesList in map(
+        lambda key: request_files.getlist(key),
+        [keys for keys in request_files])
+     for file in filesList]
+
 class TaskIDsSerializer(serializers.BaseSerializer):
     def to_representation(self, obj):
         return obj.id
@@ -26,7 +36,6 @@ class TaskSerializer(serializers.ModelSerializer):
     project = serializers.PrimaryKeyRelatedField(queryset=models.Project.objects.all())
     processing_node = serializers.PrimaryKeyRelatedField(queryset=ProcessingNode.objects.all()) 
     processing_node_name = serializers.SerializerMethodField()
-    images_count = serializers.SerializerMethodField()
     can_rerun_from = serializers.SerializerMethodField()
 
     def get_processing_node_name(self, obj):
@@ -35,9 +44,6 @@ def get_processing_node_name(self, obj):
         else:
             return None
 
-    def get_images_count(self, obj):
-        return obj.imageupload_set.count()
-
     def get_can_rerun_from(self, obj):
         """
         When a task has been associated with a processing node
@@ -142,11 +148,7 @@ def retrieve(self, request, pk=None, project_pk=None):
     def create(self, request, project_pk=None):
         project = get_and_check_project(request, project_pk, ('change_project', ))
 
-        # MultiValueDict in, flat array of files out
-        files = [file for filesList in map(
-                        lambda key: request.FILES.getlist(key), 
-                        [keys for keys in request.FILES])
-                    for file in filesList]
+        files = flatten_files(request.FILES)
 
         if len(files) <= 1:
             raise exceptions.ValidationError(detail="Cannot create task, you need at least 2 images")
@@ -157,6 +159,7 @@ def create(self, request, project_pk=None):
 
             for image in files:
                 models.ImageUpload.objects.create(task=task, image=image)
+            task.images_count = len(files)
 
             # Update other parameters such as processing node, task name, etc.
             serializer = TaskSerializer(task, data=request.data, partial=True)
@@ -198,7 +201,7 @@ def partial_update(self, request, *args, **kwargs):
 
 class TaskNestedView(APIView):
     queryset = models.Task.objects.all().defer('orthophoto_extent', 'dtm_extent', 'dsm_extent', 'console_output', )
-    permission_classes = (IsAuthenticatedOrReadOnly, )
+    permission_classes = (AllowAny, )
 
     def get_and_check_task(self, request, pk, annotate={}):
         try:
@@ -322,3 +325,43 @@ def get(self, request, pk=None, project_pk=None, unsafe_asset_path=""):
             raise exceptions.NotFound("Asset does not exist")
 
         return download_file_response(request, asset_path, 'inline')
+
+"""
+Task assets import
+"""
+class TaskAssetsImport(APIView):
+    permission_classes = (permissions.AllowAny,)
+    parser_classes = (parsers.MultiPartParser, parsers.JSONParser, parsers.FormParser,)
+
+    def post(self, request, project_pk=None):
+        project = get_and_check_project(request, project_pk, ('change_project',))
+
+        files = flatten_files(request.FILES)
+        import_url = request.data.get('url', None)
+        task_name = request.data.get('name', 'Imported Task')
+
+        if not import_url and len(files) != 1:
+            raise exceptions.ValidationError(detail="Cannot create task, you need to upload 1 file")
+
+        if import_url and len(files) > 0:
+            raise exceptions.ValidationError(detail="Cannot create task, either specify a URL or upload 1 file.")
+
+        with transaction.atomic():
+            task = models.Task.objects.create(project=project,
+                                              auto_processing_node=False,
+                                              name=task_name,
+                                              import_url=import_url if import_url else "file://all.zip",
+                                              status=status_codes.RUNNING,
+                                              pending_action=pending_actions.IMPORT)
+            task.create_task_directories()
+
+            if len(files) > 0:
+                destination_file = task.assets_path("all.zip")
+                with open(destination_file, 'wb+') as fd:
+                    for chunk in files[0].chunks():
+                        fd.write(chunk)
+
+            worker_tasks.process_task.delay(task.id)
+
+        serializer = TaskSerializer(task)
+        return Response(serializer.data, status=status.HTTP_201_CREATED)

app/api/urls.py

@@ -3,7 +3,7 @@
 from app.api.presets import PresetViewSet
 from app.plugins import get_api_url_patterns
 from .projects import ProjectViewSet
-from .tasks import TaskViewSet, TaskTiles, TaskTilesJson, TaskDownloads, TaskAssets
+from .tasks import TaskViewSet, TaskTiles, TaskTilesJson, TaskDownloads, TaskAssets, TaskAssetsImport
 from .processingnodes import ProcessingNodeViewSet, ProcessingNodeOptionsView
 from rest_framework_nested import routers
 from rest_framework_jwt.views import obtain_jwt_token
@@ -25,10 +25,9 @@
 
     url(r'projects/(?P<project_pk>[^/.]+)/tasks/(?P<pk>[^/.]+)/(?P<tile_type>orthophoto|dsm|dtm)/tiles/(?P<z>[\d]+)/(?P<x>[\d]+)/(?P<y>[\d]+)\.png$', TaskTiles.as_view()),
     url(r'projects/(?P<project_pk>[^/.]+)/tasks/(?P<pk>[^/.]+)/(?P<tile_type>orthophoto|dsm|dtm)/tiles\.json$', TaskTilesJson.as_view()),
-
     url(r'projects/(?P<project_pk>[^/.]+)/tasks/(?P<pk>[^/.]+)/download/(?P<asset>.+)$', TaskDownloads.as_view()),
-
     url(r'projects/(?P<project_pk>[^/.]+)/tasks/(?P<pk>[^/.]+)/assets/(?P<unsafe_asset_path>.+)$', TaskAssets.as_view()),
+    url(r'projects/(?P<project_pk>[^/.]+)/tasks/import$', TaskAssetsImport.as_view()),
 
     url(r'^auth/', include('rest_framework.urls')),
     url(r'^token-auth/', obtain_jwt_token),

app/migrations/0025_auto_20190220_1854.py

@@ -0,0 +1,69 @@
+# Generated by Django 2.1.5 on 2019-02-20 18:54
+
+import app.models.task
+import colorfield.fields
+import django.contrib.postgres.fields
+import django.contrib.postgres.fields.jsonb
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('app', '0024_update_task_assets'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='task',
+            name='import_url',
+            field=models.TextField(blank=True, default='',
+                                   help_text='URL this task is imported from (only for imported tasks)'),
+        ),
+        migrations.AlterField(
+            model_name='preset',
+            name='options',
+            field=django.contrib.postgres.fields.jsonb.JSONField(blank=True, default=list,
+                                                                 help_text="Options that define this preset (same format as in a Task's options).",
+                                                                 validators=[app.models.task.validate_task_options]),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='available_assets',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=80), blank=True,
+                                                            default=list,
+                                                            help_text='List of available assets to download',
+                                                            size=None),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='options',
+            field=django.contrib.postgres.fields.jsonb.JSONField(blank=True, default=dict,
+                                                                 help_text='Options that are being used to process this task',
+                                                                 validators=[app.models.task.validate_task_options]),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='pending_action',
+            field=models.IntegerField(blank=True, choices=[(1, 'CANCEL'), (2, 'REMOVE'), (3, 'RESTART'), (4, 'RESIZE'),
+                                                           (5, 'IMPORT')], db_index=True,
+                                      help_text='A requested action to be performed on the task. The selected action will be performed by the worker at the next iteration.',
+                                      null=True),
+        ),
+        migrations.AlterField(
+            model_name='theme',
+            name='header_background',
+            field=colorfield.fields.ColorField(default='#3498db', help_text="Background color of the site's header.",
+                                               max_length=18),
+        ),
+        migrations.AlterField(
+            model_name='theme',
+            name='tertiary',
+            field=colorfield.fields.ColorField(default='#3498db', help_text='Navigation links.', max_length=18),
+        ),
+        migrations.AddField(
+            model_name='task',
+            name='images_count',
+            field=models.IntegerField(blank=True, default=0, help_text='Number of images associated with this task'),
+        ),
+    ]