docs: Add llava, update hot topics (#1322)

Signed-off-by: Ettore Di Giacinto <[email protected]>

README.md

@@ -22,7 +22,7 @@
 
 > :bulb: Get help - [❓FAQ](https://localai.io/faq/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) [:book: Documentation website](https://localai.io/)
 > 
-> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/)
+> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/) [ 🚀 Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
 
 
 [![tests](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml)[![Build and Release](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml)[![build container images](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml)[![Bump dependencies](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml)[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/localai)](https://artifacthub.io/packages/search?repo=localai)
@@ -79,7 +79,12 @@ LocalAI was created by [Ettore Di Giacinto](https://github.com/mudler/) and is a
 
 Note that this started just as a [fun weekend project](https://localai.io/#backstory) in order to try to create the necessary pieces for a full AI assistant like `ChatGPT`: the community is growing fast and we are working hard to make it better and more stable. If you want to help, please consider contributing (see below)!
 
-## 🔥🔥 [Hot topics / Roadmap](https://localai.io/#-hot-topics--roadmap)
+## 🔥🔥 Hot topics / Roadmap
+
+- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
+
+Hot topics:
+- https://github.com/mudler/LocalAI/issues/1126
 
 ## 🚀 [Features](https://localai.io/features/)
 
@@ -91,7 +96,7 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs
 - 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
 - ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
 - 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
-
+- 🆕 [Vision API](https://localai.io/features/gpt-vision/)
 
 ## :book: 🎥 [Media, Blogs, Social](https://localai.io/basics/news/#media-blogs-social)
 

docs/content/_index.en.md

@@ -20,7 +20,7 @@ title = "LocalAI"
 
 > 💡 Get help - [❓FAQ](https://localai.io/faq/) [❓How tos](https://localai.io/howtos/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [💭Discord](https://discord.gg/uJAeKSAGDy)
 >
-> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/)
+> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/) [ 🚀 Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
 
 **LocalAI** is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU. It is maintained by [mudler](https://github.com/mudler).
 
@@ -84,37 +84,15 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs
 - 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
 - ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
 - 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
+- 🆕 [Vision API](https://localai.io/features/gpt-vision/)
+
 
 ## 🔥🔥 Hot topics / Roadmap
 
-- Backend rework for LocalAI v2 https://github.com/go-skynet/LocalAI/issues/1126
-- Roadmap: https://github.com/go-skynet/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap
-
-Some items:
-
-- [ ] Enable gallery management directly from the webui. https://github.com/go-skynet/LocalAI/issues/918
-- [x] llama.cpp lora adapters: https://github.com/go-skynet/LocalAI/issues/919
-- [ ] image variants and edits: https://github.com/go-skynet/LocalAI/issues/921
-- [ ] esrgan for diffusers: https://github.com/go-skynet/LocalAI/issues/917
-- [ ] ggml-stablediffusion: https://github.com/go-skynet/LocalAI/issues/916
-- [ ] SAM: https://github.com/go-skynet/LocalAI/issues/915
-- [x] diffusers lora adapters: https://github.com/go-skynet/LocalAI/issues/914
-- [ ] resource management and control: https://github.com/go-skynet/LocalAI/issues/912
-- [ ] ChatGLM: https://github.com/go-skynet/LocalAI/issues/754
-- [ ] text-to-video : https://github.com/go-skynet/LocalAI/issues/933
-- [ ] rustformers: https://github.com/go-skynet/LocalAI/issues/939
-- [x] Vall-e: https://github.com/go-skynet/LocalAI/issues/985
-- [ ] Speculative sampling: https://github.com/go-skynet/LocalAI/issues/1013
-- [ ] Falcon/GPTNeoX on llama.cpp: https://github.com/go-skynet/LocalAI/issues/1009
-- [ ] transformers/vllm: https://github.com/go-skynet/LocalAI/issues/1015
-- [ ] TortoiseTTS: https://github.com/go-skynet/LocalAI/issues/1016
-- [ ] Exllama2: https://github.com/go-skynet/LocalAI/issues/1053
-- [ ] ctransformers: https://github.com/go-skynet/LocalAI/issues/1056
-- [ ] GPTQ for LLama: https://github.com/go-skynet/LocalAI/issues/1055
-- [ ] LLaVA and miniGPT-4: https://github.com/go-skynet/LocalAI/issues/1054
-- [ ] Test generation inference: https://github.com/go-skynet/LocalAI/issues/1042
-- [ ] docs - extending LocalAI with external backends: https://github.com/go-skynet/LocalAI/issues/1057
+- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
 
+Hot topics:
+- https://github.com/mudler/LocalAI/issues/1126
 
 ## How does it work?
 

docs/content/features/_index.en.md

@@ -13,4 +13,5 @@ This section contains the documentation for the features supported by LocalAI.
 - [🎨 Image generation]({{%relref "features/image-generation" %}})
 - [🧠 Embeddings]({{%relref "features/embeddings" %}})
 - [🔥 OpenAI functions]({{%relref "features/openai-functions" %}})
+- [🆕 GPT Vision API]({{%relref "features/gpt-vision" %}})
 - [✍️ Constrained grammars]({{%relref "features/constrained_grammars" %}})

docs/content/features/gpt-vision.md

@@ -0,0 +1,30 @@
+
++++
+disableToc = false
+title = "🆕 GPT Vision"
+weight = 2
++++
+
+{{% notice note %}}
+Available only on `master` builds
+{{% /notice %}}
+
+LocalAI supports understanding images by using [LLaVA](https://llava.hliu.cc/), and implements the [GPT Vision API](https://platform.openai.com/docs/guides/vision) from OpenAI.
+
+![llava](https://github.com/mudler/LocalAI/assets/2420543/cb0a0897-3b58-4350-af66-e6f4387b58d3)
+
+## Usage
+
+OpenAI docs: https://platform.openai.com/docs/guides/vision
+
+To let LocalAI understand and reply with what sees in the image, use the `/v1/chat/completions` endpoint, for example with curl:
+
+```bash
+curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
+     "model": "llava",
+     "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'
+```
+
+### Setup
+
+To setup the LLaVa models, follow the full example in the [configuration examples](https://github.com/mudler/LocalAI/blob/master/examples/configurations/README.md#llava).

docs/content/model-compatibility/vall-e-x.md

@@ -1,7 +1,7 @@
 
 +++
 disableToc = false
-title = "🆕 Vall-E-X"
+title = "Vall-E-X"
 weight = 4
 +++
 

docs/content/model-compatibility/vllm.md

@@ -1,7 +1,7 @@
 
 +++
 disableToc = false
-title = "🆕 vLLM"
+title = "vLLM"
 weight = 4
 +++