Skip to content

adiog/io_quicksave_dev

Repository files navigation

quicksave

Running quicksave locally

Requirements

  • screen
  • docker
  • python3.6 and virtualenv
  • nginx

note: it is not safe, but if you want to avoid problems with running multiple screen sessions, you may modify your /etc/sudoers file:

youruser ALL=(ALL) NOPASSWD:ALL

What is the impact on the host system?

Test environment is intended to be run locally, so the following /etc/hosts entries will be placed:

127.0.0.1 api.quicksave.io cdn.quicksave.io oauth.quicksave.io www.quicksave.io quicksave.io

These changes will be automatically made by ./bootstrap.sh script.

Additionally the nginx sites associated with these domains will be registered. By default the following ports will be used:

8080 ~ quicksave.io
11000 ~ api.quicksave.io
12000 ~ cdn.quicksave.io
13000 ~ oauth.quicksave.io
13666 ~ audit.quicksave.io

The configuration can be found under https://github.com/adiog/io_quicksave_dev/tree/master/etc

Bootstrapping

Just run:

$ ./bootstrap.sh

WARNING: sudo will be used with each container (you may have to check all 10 screen sessions! ($ screen -r qs<TAB>)).

The STUPID option:

$ ./suicide.sh && ./deps.sh && ./bootstrap.sh

Usage

Command line tool

If you do not want to pollute your python environment activate virtualenv:

. ~/.quicksave/venv/bin/activate

The command line tool qs will be added to your PATH:

$ qs --text \"do homework\"
$ qs --file VeryImportant.pdf
$ qs --clipboard
$ qs --screenshot
$ qs --input

Please mind that some additional dependancies are implied (e.g. gnome-screenshot, gedit, xsel). The configuration file can be found under:

~/.quicksave/quicksave.ini

If you want to bind a qs command to a keyboard shortcut, you may use the --gui switch for x-window credentials prompt. E.g.:

qs --gui --area

The default session length is set to 1 hour.

Note: to use commandline client system-wise:

sudo pip3.6 install git+https://github.com/adiog/io_quicksave_client --upgrade

QSQL queries

Currently the basic filtering can be done only with QSQL, the syntax is straighforward:

WHERE name = 'exact match'
WHERE name ~ 'pattern'
WHERE tag 'yourtag'
WHERE ((name ~ 'Screenshot') OR (tag 'funpic'))

Note: excessive parenthesis are currently mandatory.

Known Issues

There is a lot..

Design notes

Overview

'quicksave system' provides an easy-access online 'database'.

'objects' can have a different form:

  • freetext (selection, content of clipboard etc.)
  • file (pdf, image, screenshot, etc.)
  • link to online resource (page, image, media, etc.) etc.

the actual behaviour of creating and retrieving objects is determined by configurable set of plugins.

Object passing

Bean stands here for an serializable object satisfying a json signature. It is a simply ABI specification. It will be renamed to avoid confusion. Or even replaced by existing solutions like protobuf.

Bean objects used by system can be found in beans repository.

libbeans repository provides a collection of generators and libraries:

  • python module
  • c++ headers
  • c++ bean generator
  • c++ database generator (currently sqlite)
  • sql generator ('create migration')

Components

www

static files. html, javascript and css

main dependencies:

  • jquery
  • semantic-ui

other dependencies

  • mathjax
  • prettyprint

www component can be found in www repository

qsql

quicksave query language. c++, python and javascript parsers

main dependencies:

  • antlr

qsql component can be found in qsql repository

api / oauth / cdn

c++ backend.

main dependencies:

  • proxygen
  • rapidjson
  • sqlite/postgres

backend components can be found in cppapi repository

remark: oauth does not do actual OAuth v2 protocol. It simply authenticates the user and emits a token / creates the token/session/context entry in global memcached service.

post

c++ backend collecting the tasks from python async via rabbitmq queue.

dependencies:

  • rabbitmq-c

remark: this component is intentionally extracted to prevent QM Python code from directly accessing a db.

EXPERIMENTAL fuse adapter

simple c++ adpater exposing the result of QSQL query directly as static in-memory fuse filesystem.

plugin-engine

python-based plugin execution framework.

consisted of three main components:

  • syncronous engine (initial plugin execution; detect type; determine longer tasks)
  • asynchronous engine (longer tasks - e.g. getting thumbnail, backuping resources etc.)
  • javascript front-end (presentation layer)

plugin-engine component can be found in plugin-engine repository

Note: the single plugin-engine repository is deprecated. The following ones will be used io_quicksave_api-ext (safe python), io_quicksave_async (unsafe python), io_quicksave_www (javascript).

browser plugin / social API / bookmarklet

browser integration.

chrome plugin can be found in chrome plugin repository firefox plugin can be found in firefox plugin repository

Firefox Add-on has been already published under firefox plugin

command line client

python-based command line client and explorer integration.

cli client can be found in cli client repository

Architecture

alt text

Bottlenecks

  • master database (used once on authentication)
  • memcached (one instance - cannot be replaced by a raw copy)
  • rabbitmq

Horizontal scaling

other components thanks to independance can be used without any restriction on number of instances.

User owned parts

every user should provide his own database and storage service. of course all metadata can be store in a single database, but this is not a constraint. Currently there are only two storage variants supported: local and sshfs. it can be easily extend to cover gdrive/dropbox. There is no GUI for signups. But feel free to test it locally.

CreateRequest

alt text

API Drafts

swagger.quicksave.io

About

entry point: bootstrap.sh

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published