From a4b7f8e3a0421b5c0b435a4d17332af4523883d1 Mon Sep 17 00:00:00 2001 From: Dustin Franklin Date: Wed, 7 Aug 2024 00:54:12 -0400 Subject: [PATCH] added Gapi --- docs/research.md | 28 ++++-- docs/tutorial-intro.md | 9 +- docs/tutorial_gapi_microservices.md | 84 ++++++++++++++++ docs/tutorial_gapi_workflows.md | 151 ++++++++++++++++++++++++++++ mkdocs.yml | 3 + 5 files changed, 265 insertions(+), 10 deletions(-) create mode 100644 docs/tutorial_gapi_microservices.md create mode 100644 docs/tutorial_gapi_workflows.md diff --git a/docs/research.md b/docs/research.md index 5a961f8e..6e48f946 100644 --- a/docs/research.md +++ b/docs/research.md @@ -10,9 +10,9 @@ The Jetson AI Lab Research Group is a global collective for advancing open-sourc There are virtual [meetings](#meeting-schedule) that anyone is welcome to join, offline discussion on the [Jetson Projects](https://forums.developer.nvidia.com/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78){:target="_blank"} forum, and guidelines for upstreaming open-source [contributions](#contribution-guidelines). -!!! abstract "Next Meeting - 8/6" +!!! abstract "Next Meeting - 8/20" - The next team meeting is on Tuesday, August 6th at 9am PST - see the [invite](#meeting-schedule) below or click [here](https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTA4ZmE4MDAtYWUwMS00ZTczLWE0YWEtNTE5Y2JkNTFmOWM1%40thread.v2/0?context=%7b%22Tid%22%3a%2243083d15-7273-40c1-b7db-39efd9ccc17a%22%2c%22Oid%22%3a%221f165bb6-326c-4610-b292-af9159272b08%22%7d){:target="_blank"} to join the meeting in progress. + The next team meeting is on Tuesday, August 20th at 9am PST - see the [invite](#meeting-schedule) below or click [here](https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTA4ZmE4MDAtYWUwMS00ZTczLWE0YWEtNTE5Y2JkNTFmOWM1%40thread.v2/0?context=%7b%22Tid%22%3a%2243083d15-7273-40c1-b7db-39efd9ccc17a%22%2c%22Oid%22%3a%221f165bb6-326c-4610-b292-af9159272b08%22%7d){:target="_blank"} to join the meeting in progress. ## Topics of Interest @@ -75,7 +75,7 @@ Ongoing technical discussions are encouraged to occur on the forums or GitHub Is We'll aim to meet monthly or bi-weekly as a team in virtual meetings that anyone is welcome to join and speak during. We'll discuss the latest updates and experiments that we want to explore. Please remain courteous to others during the calls. We'll stick around after for anyone who has questions or didn't get the chance to be heard. -!!! abstract "Tuesday August 6th at 9am PST (8/6/24)" +!!! abstract "Tuesday August 20th at 9am PST (8/6/24)" - Microsoft Teams - [Meeting Link](https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTA4ZmE4MDAtYWUwMS00ZTczLWE0YWEtNTE5Y2JkNTFmOWM1%40thread.v2/0?context=%7b%22Tid%22%3a%2243083d15-7273-40c1-b7db-39efd9ccc17a%22%2c%22Oid%22%3a%221f165bb6-326c-4610-b292-af9159272b08%22%7d){:target="_blank"} - Meeting ID: `264 770 145 196` @@ -102,6 +102,20 @@ The agenda will be listed here beforehand - post to the forum to add agenda item
August 6, 2024 +
+ +

Topics Covered:

+ +
    +
  • OpenVLA Fine-Tuning
    • +
  • Gemma-2-2b (Asier Arranz)
    • +
  • Ultralytics YOLOv8 (Lakshantha Dissanayake)
    • +
+ +
+ +
July 23, 2024 +

Topics Covered:

@@ -630,13 +644,13 @@ Below are shown some of the sustaining members of the group who have been workin
-
- +
+ - Kerry Shih, OStream
+ Kerry Shih, GenAI Nerds
Founder, CEO | Los Angeles, CA
- (GenRunner, GenAI Nerds) + (Gapi)
diff --git a/docs/tutorial-intro.md b/docs/tutorial-intro.md index b8426fb0..eb7f09fe 100644 --- a/docs/tutorial-intro.md +++ b/docs/tutorial-intro.md @@ -22,7 +22,6 @@ Give your locally running LLM an access to vision! | | | | :---------- | :----------------------------------- | -| **[Mini-GPT4](./tutorial_minigpt4.md)** | [Mini-GPT4](https://minigpt-4.github.io/), an open-source model that demonstrate vision-language capabilities.| | **[LLaVA](./tutorial_llava.md)** | [Large Language and Vision Assistant](https://llava-vl.github.io/), multimodal model that combines a vision encoder and LLM for visual and language understanding. | | **[Live LLaVA](./tutorial_live-llava.md)** | Run multimodal models interactively on live video streams over a repeating set of prompts. | | **[NanoVLM](./tutorial_nano-vlm.md)** | Use mini vision/language models and the optimized multimodal pipeline for live streaming. | @@ -61,12 +60,16 @@ Give your locally running LLM an access to vision! | **[AudioCraft](./tutorial_audiocraft.md)** | Meta's [AudioCraft](https://github.com/facebookresearch/audiocraft), to produce high-quality audio and music | | **[Voicecraft](./tutorial_voicecraft.md)** | Interactive speech editing and zero shot TTS | - -## NVIDIA SDKs +### NVIDIA SDKs | | | | :---------- | :----------------------------------- | | **[Holoscan SDK](./tutorial_holoscan.md)** | Use the [Holoscan-SDK](https://github.com/nvidia-holoscan/holoscan-sdk) to run high-throughput, low-latency edge AI pipelines | +### Gapi / API Integrations +| | | +| :---------- | :----------------------------------- | +| **[Workflows](./tutorial_gapi_workflows.md)** | Integrating generative AI into real world environments | +| **[Micro Services](./tutorial_gapi_microservices.md)** | Wrapping models and code to participate in systems | ## About NVIDIA Jetson diff --git a/docs/tutorial_gapi_microservices.md b/docs/tutorial_gapi_microservices.md new file mode 100644 index 00000000..d9c070fd --- /dev/null +++ b/docs/tutorial_gapi_microservices.md @@ -0,0 +1,84 @@ +# Tutorial - Gapi Micro Services + +A Micro Service is a process that runs a wrapper python script that integrates your custom code/models so they can integrate into Gapi Workflows. + + + +You can run a Micro Service wherever you like and connect it to a Gapi Server via the streaming, hybrid binary+json message protocol. + + + +There are some out-of-the-box โ€œCommunity Micro Services" that we integrate, test and pack into Docker images. +When you run them, they auto integrate, load NVIDIA layers correctly and offer logging to the host system. + + + +Even better, create your own! +Just implement an **on_message** Python handler to process requests and respond. The rest is handled for you. + + + +Think of a Micro Service as a simple wrapper to some code or model you have. +It works the same as any other Node inside a Workflow. When it's your Micro Service Node's turn your **on_message** function will be called. +Your script gets the rolling Transaction data for context and you then publish your data directly back into the flow. + + + +## **Running the Community Micro Services** + +!!! abstract "Requirements for Community Micro Services" + + 1. One of the following Jetson devices: + + Jetson AGX Orin (64GB) + Jetson AGX Orin (32GB) + Jetson Orin NX (16GB) + Jetson Orin Nano (8GB) + Jetson Orin Nano (4GB) + + 2. Running one of the following versions of [JetPack](https://developer.nvidia.com/embedded/jetpack){:target="_blank"}: + + JetPack 5 (L4T r35.x) + JetPack 6 (L4T r36.x) + + 3. Sufficient storage space. + - Literally 4KB for your own Micro Service + - Anywhere from ~4GB to ~10GB for each Community Micro Service + +``` +#1 Login and go to the Micro Services tab +#2 Follow the instructions in the blue box on that page to download your custom configuration +#3 Then follow the instructions below that for installing the Micro Service you want +``` + +Example of instruction page: + + + +## **Congrats! You Can Go Through the Workflow Tips Now** + + + +## **Creating Your Own Micro Service** + +The entire Micro Service zip file is just 4KB with 4 files: + +* message_handler.py: for you to respond +* message.py: for the streaming binary/json protocol +* gapi-ms: as entry point and handler) +* requirements.txt: defines just asyncio + websockets + +Full documentation here: [Gapi Micro Service Docs](https://genainerds.com/#/Docs/14){:target="_blank"}. Synopsis below... + +``` +#1 Create logical Micro Service in UI and copy the key +#2 Download the zip file from the UI +#3 python gapi-ms.py ws://0.0.0.0:8090/gapi-ws [MICROSERVICE_KEY] +#4 Refresh the UI to confirm it's online +#5 Edit the message_handler.py to handle binary+json input and change the output +#6 Add a Micro Service Node to a Workflow and tie it to your Micro Service. Hit Test. +``` + + + + diff --git a/docs/tutorial_gapi_workflows.md b/docs/tutorial_gapi_workflows.md new file mode 100644 index 00000000..137275fc --- /dev/null +++ b/docs/tutorial_gapi_workflows.md @@ -0,0 +1,151 @@ +# Tutorial - Gapi + +[Gapi](https://GenAINerds.com/#/Gapi){:target="_blank"} is an embeddable API gateway that creates streaming integrations between +AI micro services and the systems that users leverage everyday. + +!!! admonition "The project's goal is to accelerate the speed of creating pilots and demos of Jetson AI Lab achievements into real world environments" + +"On Device" generative AI doesn't mean it has to live on an island! + + + +* Workflow engine with low code UI with dozens of open integrations and customizable clients for mobile web and desktop. +* Micro service framework for wrapping Jetson containers (Ollama, Whisper, Piper TTS, etc. are done, with more coming). +Or wrap your own models/code and integrate it into Workflows. +* Real-time, hybrid, binary+json messaging smoothens intra-service calls and reduced latency. +* A fast path to proving generative AI value to stakeholders in their actual environment. + +## **Gapi Server** +Embeddable API gateway software that runs in the background with a low code workflow UI for testing. +The server is a message hub and state machine for workflow 'nodes' that talk to Micro Services. +Think of it as connective-tissue for applications. + + + +!!! admonition "A Micro Service is a process that runs some wrapper python scripts that integrates custom code/models into Workflows using a streaming API." + +* Gapi Server can run on any Jetson Orin or really any computer as the Micro Services connect outbound over secure web sockets. It +doesn't use any GPU resources. There is a also a little demo version to skip the Server install +(but you'll still need to run your own Micro Services). + +* **[Gapi Project Page](https://GenAINerds.com/#/Gapi){:target="_blank"}** +* **[Gapi Github](https://github.com/GenAI-Nerds/Gapi/){:target="_blank"}** +* **[Gapi Docs](https://genainerds.com/#/Docs){:target="_blank"}** +* **[Gapi Hosted Demo](https://GenAIGapi.com){:target="_blank"}** + +!!! abstract "What you need to run Gapi Server on Jetson" + + 1. One of the following Jetson devices: + + Jetson AGX Orin (64GB) + Jetson AGX Orin (32GB) + Jetson Orin NX (16GB) + Jetson Orin NX (8GB) + Jetson Orin Nano (8GB) + Jetson Orin Nano (4GB) + + 2. Docker + + 3. Sufficient storage space. + - Size: ~1.3GB + +!!! admonition "Gapi Server will run on other environments. Email us at support@GenAINerds.com if that's something you think is worthwhile." + +Explaining the Steps: + +- 1) On the Docker host, create working dir for persistant data +- 2) Download configuration files +- 3) Unzip +- 4) Pull Docker image, create container and start the process (will return console to you) + +Copy and Run the Steps: + +``` +mkdir ~/gapiData && cd ~/gapiData +curl -L https://raw.githubusercontent.com/genai-nerds/Gapi/main/gapiConfigs.zip -o gapiConfigs.zip +unzip -q gapiConfigs.zip +docker run -d --name gapi --network host -v ~/gapiData:/opt/gapi/vdata genainerds/gapi:arm64 /bin/bash -c "cd /opt/gapi/bin && ./startGapi.sh" +echo "You may need to hit Enter now. Afterwards the Docker container 'gapi' should be running" +``` + +Troubleshooting: + +- Keep in mind all data read or written is in ~/gapiData +- Look at ~/gapiData/gapi.log to see what happened (if say the docker run command doesn't work) +- gapiServerConfig.json has all the initial setup + +!!!admonition "NOTE: You will need to run some Micro Services before doing anything meaningful, so please review the mini tour below but don't do any of it in the UI untill you complete the setup (instructions at the bottom)" + +## **UI** + + +* Browse in: http://[host-device-ip]:8090 +* User: root +* Pass: !gapi2024 + +* Change password in Settings! Docs shows how to add SSL cert. + +## **Tips & Use Case Templates** + +When you login there will be an array of Tip Workflows that have notes and explain core concepts. + +**Tips:** + +* Hello World: Basics plus it literally says hello +* Run a Local LLM: Play with Ollama graphically and connect it to other systems +* Streaming Speech to Text: PiperTTS +* Querying a Vector Database: Query a simple set of vectorized text documents +* Variables, Flow and Logic: Understand how to setup more robust workflows +* Calling Workflows from Outside Gapi: Configure Webhooks +* Workflows Calling Your Code: Micro Service Nodes that invoke your code +* Communications: 3rd party communications like Slack (IM), Twilio (SMS), SendGrid (EMAIL) + +## **Workflows** + +Workflows visually connect the execution and flow of data between Nodes. + + +A Transaction (or single firing) has "Rolling Input" data it accumulates as Node to Node steps each talk to Micro Services and APIs. +All Nodes enjoy variables and flow control using familiar json and javascript concepts. + + + +Each Node can append or reference the rolling data in the Transaction while making decisions along the way. + + + +Watch live Transactions as they start from clients, webhooks and published messages from Micro Services with visual feedback and debugging. + +## **APIs to Business Systems** + +Gapi can help smoothly integrate generative AI into systems that people already use everyday via APIs. +It has the streaming API to Micro Services plus the state management and chops to handle the outward (webhook) style APIs to existing +systems. + +!!!admonition "Our hope is to rally contributions by the community to keep growing the out-of-the-box/tested Nodes but there is a DIY one as well to manually map what you need into your Workflows." + +Some of the out-of-the-box API Nodes: Slack (IM), Twilio (SMS), SendGrid (Email), Service Now (Ticketing), DIY Webhook + +## **Micro Services** + +There are community published Micro Services as well as custom ones you can make yourself. +Gapi Server becomes most useful when leveraging them so please follow the How To below. + +**Current Community Micro Services:** + +* Whisper +* Ollama +* Vector +* Text to Speech +* Img to Text + +!!! admonition "Complete the Setup: [How To Run and/or Create Micro Services](./tutorial_gapi_microservices.md)" + +## **Support / Contribute** + +Gapi is a project from the [GenAI Nerds](https://GenAINerds.com) and hosted on [Github](https://github.com/GenAI-Nerds/Gapi/){:target="_blank"}. + +* Ask a question, support@GenAINerds.com or +* Say hello, hello@GenAINerds.com +* Contribute/create tickets on [Github](https://github.com/GenAI-Nerds/Gapi/){:target="_blank"} + diff --git a/mkdocs.yml b/mkdocs.yml index 45cedbd3..9fab5a42 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -113,6 +113,9 @@ nav: - Stable Diffusion XL: tutorial_stable-diffusion-xl.md - NVIDIA SDKs: - Holoscan SDK ๐Ÿ†•: tutorial_holoscan.md + - Gapi / API Integrations: + - Workflows ๐Ÿ†•: tutorial_gapi_workflows.md + - Micro Services ๐Ÿ†•: tutorial_gapi_microservices.md #- Metropolis Microservices: # - First Steps: tutorial_mmj.md # - Tools: