The search index offers the possibility to search quickly in various oer sources (mainly in the context of higher education).
With this project you can set up all components that are necessary to run the index. The process uses ansible to install the components.
You can find stable versions of the setup in the branch master and also tagged fixed releases - see https://gitlab.com/oersi/oersi-setup/-/releases.
As a link between the individual OER repositories, the index harvests (e.g. daily) the repositories of the federal states, or generally data sources that provide metadata, such as the AV portal, SlideWiki etc.
The same index can also be reused at the level of a federal state (or generally distributed), assuming that no data storage is required, but only linking of content (e.g. from LMS or GitLab). Here the harvester can be used to read out e.g. the LMS of the universities or a state-wide GitLab instance. Alternatively, a user management system can be connected for this purpose, with which individual persons can insert, change or delete their own content.
- ETL: The ETL-module (extract, transform, load) connects to the individual repositories and fetches metadata updates according to a configured schedule (e.g. daily).
- Import Scripts: Another channel to connect individual OER repositories and import metadata according to a configured schedule (e.g. daily). Contains scripted imports (e.g. Python)
- API / Backend: Provides interfaces to retrieve data from the index (external) and to import / update data into the index (internal). A read-only user is used to retrieve data. When creating / updating the data, the data is written into an elasticsearch index.
- Frontend: Website that displays the data from the index and offers the possibility to search
- User Management: (TDB) Optional - only needed if the index is used as a (local) standalone solution. So users can be given access to manually update the data via a UI.
The latest stable version is available in the master-branch. There are also tagged stable versions (0.7.0 and later).
The installation processes will install all components that are necessary to run oersi (sql-db, elasticsearch, tomcat, nginx, etc). To avoid conflicts, it is recommended that no other applications / processes are running on the target system.
Depending on the scenario, there are different possibilities to install oersi. To test the index a little bit with minimal effort, you can use a local installation in the local VirtualBox VM via vargant. The other installation methods are suitable for installing the index on an existing system that is accessible via ansible. Of these, the installation via gitlab-ci is clearly recommended.
Note that all installations (especially the initial one) will take some time (depending on your download speed), because all necessary components will be downloaded.
Scenario: Install oersi on an existing system. Can be controlled via the UI of gitlab and can even be started on a scheduled basis. In addition, the environment from which the installation is performed is always the same, since this is done via docker containers, and you don't run into conflicts due to different versions of Ansible on the installation host. This makes it more suitable for installations on production systems, as the installation becomes more stable. This process is the integration of the "normal" installation process (via ansible-playbook) into the CI of gitlab.
- Create a new private gitlab project that will contain the configuration for your oersi-systems
- Assure your gitlab-project is able to connect to your oersi-systems via ssh
- Create an ssh-key for the automatic installation via gitlab
- Your public ssh-key needs to be added to the authorized_keys file on your oersi-systems
vi ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys- go to Settings -> CI / CD -> Variables of your project in gitlab and add variables
- SSH_KNOWN_HOSTS - contains entries for your oersi-systems
- SSH_PRIVATE_KEY - your private ssh-key
- Use .gitlab-ci.yml and prerequisites.yml from the gitlab-config-example
- Create your own inventory-files and config-files with help of inventory_DEV.yml, config_DEV.yml, inventory_TEST.yml and config_TEST.yml
- When you adjust the filenames of your files, please assure to also adjust the filenames in .gitlab-ci.yml
- Assure root access via ansible connection variables like ansible_become_password.
- Now you should be able to update your systems via your gitlab-project
- Call: CI / CD -> Pipelines -> Latest
- Click the "Play"-Button for the system you want to update
- Optionally you can activate Continuous Delivery
- Create a Gitlab-Schedule in your project CI / CD -> Schedules
- uncomment "scheduled deployment" section in .gitlab-ci.yml and adjust to your inventory-filename
Scenario: To test the index a little bit with minimal effort, you can use a local installation in the local VirtualBox VM via vagrant. This is also suitable for developers to perform "system tests" for their changes.
Prerequisites
- Git (tested with 2.17.1)
- Vagrant (tested with 2.0.2)
- VirtualBox (tested with 5.2.34_Ubuntu)
- Depending on your setup, you might require a file
/etc/vbox/networks.confcontaining:
* 10.0.0.0/8 192.168.0.0/16
* 2001::/64
Perform the following steps in the terminal (Linux / macOS) or in the GitBash (Windows).
git clone https://gitlab.com/oersi/oersi-setup.git
cd oersi-setup
vagrant up
When the installation is complete (a few minutes, depending on the download speed), the index can be opened in the browser
You can stop the virtual machine with
vagrant halt
... and start again with
vagrant up
If you want to reload the configuration (including the import from the oer sources), perform
vagrant reload --provision
The VM can then simply be thrown away at any time with vagrant destroy.
Scenario: Install oersi on an existing system directly via the ansible-playbook. Can be used if installation via gitlab is not possible (e.g. because no internal gitlab instance exists and the oersi system is not accessible from gitlab.com). Note for developers: this method is also suitable for installations on the VirtualBox VM. Since you can work with your own inventory file, you can test the complete custumizations of your production environment on the VM without having to make changes in the group_vars of the Vagrant Box.
- requirements:
- ansible installed on the local computer
- clone project
- install ansible galaxy dependencies:
ansible-galaxy install -r requirements.yml
- create ansible inventory and config files (see gitlab-config-example for an example) and adjust all variables to your installation (see variables in ansible/playbooks/group_vars)
- run
ansible-playbook -v -i inventory.yml --extra-vars @default-config.yml --extra-vars @config.yml search_index.setup.system- you can also install single modules via
ansible-playbook -v -i inventory.yml --extra-vars @default-config.yml --extra-vars @config.yml search_index.setup.<MODULE>
- you can also install single modules via
See WEBANALYTICS.md
Contributions are welcome. Please read CONTRIBUTING.md for further details.
- Elasticsearch: Elasticsearch is the search engine and indexes the input it gets from the backend.
- Kibana: For development or monitoring of Elasticsearch Kibana can be installed optionally (activate in ansible/group_vars -> kibana_install)
- Backend: The backend is the interface for read/write access to the index, see https://gitlab.com/oersi/sidre/sidre-backend
- Frontend: The frontend is used from https://gitlab.com/oersi/sidre/sidre-frontend
- ETL: The harvester component uses the oersi-etl (Extract, Transform, Load) project, see https://gitlab.com/oersi/oersi-etl
- Import Scripts - Uses scripts like python to process an import of an OER repo, see https://gitlab.com/oersi/oersi-import-scripts
This project is licensed under the MIT License - see the LICENSE file for details.
Inspired by OERhoernchen 2.0 (by Matthias Andrasch) and Docker-Hoernchen 2.0 (by Steffen Rörtgen).