Merge pull request #11 from tokk-nv/main

Add SAM, TAM and NanoOWL page
NVIDIA-AI-IOT · Oct 11, 2023 · 518cfc5 · 518cfc5
2 parents 6467df1 + 51225ff
commit 518cfc5
Show file tree

Hide file tree

Showing 12 changed files with 225 additions and 5 deletions.
diff --git a/docs/images/TAM_15s_1080p.mp4 b/docs/images/TAM_15s_1080p.mp4
diff --git a/docs/images/TAM_screencast_cat_720p-80pcnt.mp4 b/docs/images/TAM_screencast_cat_720p-80pcnt.mp4
diff --git a/docs/images/TAM_screenshot.png b/docs/images/TAM_screenshot.png
diff --git a/docs/images/TAM_screenshot_cat.png b/docs/images/TAM_screenshot_cat.png
diff --git a/docs/images/sam_notebook.png b/docs/images/sam_notebook.png
diff --git a/docs/tutorial_nanodb.md b/docs/tutorial_nanodb.md
@@ -0,0 +1,4 @@
+# Tutorial - NanoDB
+
+
+
diff --git a/docs/tutorial_nanoowl.md b/docs/tutorial_nanoowl.md
@@ -0,0 +1,3 @@
+# Tutorial - NanoOWL 
+
+Check out the GitHub repo, [https://github.com/NVIDIA-AI-IOT/nanoowl](https://github.com/NVIDIA-AI-IOT/nanoowl).
diff --git a/docs/tutorial_nanosam.md b/docs/tutorial_nanosam.md
@@ -1,7 +1,78 @@
 # Tutorial - NanoSAM
 
-See "NanoSAM" repo.
+Let's run NVIDIA's [NanoSAM](https://github.com/NVIDIA-AI-IOT/nanosam) to check out the performance gain by distillation.
 
-[https://github.com/NVIDIA-AI-IOT/nanosam](https://github.com/NVIDIA-AI-IOT/nanosam)
+![](https://raw.githubusercontent.com/NVIDIA-AI-IOT/nanosam/main/assets/basic_usage_out.jpg)
 
-![](https://raw.githubusercontent.com/NVIDIA-AI-IOT/nanosam/main/assets/mouse_gif_compressed.gif)
+!!! abstract "What you need"
+
+    1. One of the following Jetson:
+
+        <span class="blobDarkGreen4">Jetson AGX Orin 64GB</span>
+        <span class="blobDarkGreen5">Jetson AGX Orin (32GB)</span>
+        <span class="blobLightGreen4">Jetson Orin Nano Orin (8GB)</span>
+
+    2. Running one of the following [JetPack.5x](https://developer.nvidia.com/embedded/jetpack)
+
+        <span class="blobPink1">JetPack 5.1.2 (L4T r35.4.1)</span>
+        <span class="blobPink2">JetPack 5.1.1 (L4T r35.3.1)</span>
+        <span class="blobPink3">JetPack 5.1 (L4T r35.2.1)</span>
+
+    3. Sufficient storage space (preferably with NVMe SSD).
+
+        - `6.3GB` for container image
+        - Spaces for models
+
+## Set up a container for `nanosam`
+
+### Clone `jetson-containers`
+
+!!! tip ""
+
+    See [`jetson-containers`' `nanosam` package README](https://github.com/dusty-nv/jetson-containers/tree/master/packages/vit/nanosam) for more infomation**
+
+```
+git clone https://github.com/dusty-nv/jetson-containers
+cd jetson-containers
+sudo apt update; sudo apt install -y python3-pip
+pip3 install -r requirements.txt
+```
+
+## How to start
+
+Use `run.sh` and `autotag` script to automatically pull or build a compatible container image.
+
+```
+cd jetson-containers
+./run.sh $(./autotag nanosam)
+```
+
+## Run examples
+
+Inside the container, you can move to `/opt/nanosam` directory, to go through all the examples demonstrated on the repo.
+
+```
+cd /opt/nanosam
+```
+
+To run the "[**Example 1 - Segment with bounding box**](https://github.com/NVIDIA-AI-IOT/nanosam#example-1---segment-with-bounding-box)":
+
+```
+python3 examples/basic_usage.py \
+    --image_encoder="data/resnet18_image_encoder.engine" \
+    --mask_decoder="data/mobile_sam_mask_decoder.engine"
+```
+
+The result is saved under `/opt/nanosam/data/basic_usage_out.jpg`.
+
+To check on your host machine, you can copy that into `/data` directory of the container where that is mounted from the host.
+
+```
+cp data/basic_usage_out.jpg /data/
+```
+
+Then you can go to your host system, and find the file under the `jetson_containers`' `data` directory, like `jetson_containers/data/basic_usage_out.jpg`.
+
+## Results
+
+![](https://raw.githubusercontent.com/NVIDIA-AI-IOT/nanosam/main/assets/basic_usage_out.jpg)
diff --git a/docs/tutorial_sam.md b/docs/tutorial_sam.md
@@ -0,0 +1,72 @@
+# Tutorial - SAM (Segment Anything)
+
+Let's run Meta's [`SAM`](https://github.com/facebookresearch/segment-anything) on NVIDIA Jetson.
+
+![](./images/sam_notebook.png)
+
+!!! abstract "What you need"
+
+    1. One of the following Jetson:
+
+        <span class="blobDarkGreen4">Jetson AGX Orin 64GB</span>
+        <span class="blobDarkGreen5">Jetson AGX Orin (32GB)</span>
+        <span class="blobLightGreen4">Jetson Orin Nano Orin (8GB)</span>⚠️[^1]
+
+    2. Running one of the following [JetPack.5x](https://developer.nvidia.com/embedded/jetpack)
+
+        <span class="blobPink1">JetPack 5.1.2 (L4T r35.4.1)</span>
+        <span class="blobPink2">JetPack 5.1.1 (L4T r35.3.1)</span>
+        <span class="blobPink3">JetPack 5.1 (L4T r35.2.1)</span>
+
+    3. Sufficient storage space (preferably with NVMe SSD).
+
+        - `6.8GB` for container image
+        - Spaces for models
+
+    [^1]: The biggest `vit_h` (2.4GB) model may not ran due to OOM, but `vit_l` (1.1GB) runs on Jetson Orin Nano.
+
+## Set up a container for `sam`
+
+### Clone `jetson-containers`
+
+!!! tip ""
+
+    See [`jetson-containers`' `sam` package README](https://github.com/dusty-nv/jetson-containers/tree/master/packages/vit/sam) for more infomation**
+
+```
+git clone https://github.com/dusty-nv/jetson-containers
+cd jetson-containers
+sudo apt update; sudo apt install -y python3-pip
+pip3 install -r requirements.txt
+```
+
+## How to start
+
+Use `run.sh` and `autotag` script to automatically pull or build a compatible container image.
+
+```
+cd jetson-containers
+./run.sh $(./autotag sam)
+```
+
+The container has a default run command (`CMD`) that will automatically start the Jupyter Lab server.
+
+Open your browser and access `http://<IP_ADDRESS>:8888`.
+
+> The default password for Jupyter Lab is `nvidia`.
+
+## Run Jupyter notebook
+
+In Jupyter Lab, navigate to `notebooks` and open `automatic_mask_generator_example.py` notebook.
+
+Create a new cell at the top, insert the model download command below and run the cell.
+
+```
+!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
+```
+
+Then go through executing all the cells below **Set-up**.
+
+## Results
+
+![](./images/sam_notebook.png)
diff --git a/docs/tutorial_tam.md b/docs/tutorial_tam.md
@@ -0,0 +1,65 @@
+# Tutorial - SAM (Segment Anything)
+
+Let's run [`TAM`](https://github.com/gaomingqi/Track-Anything) to perform Segment Anything on videos on NVIDIA Jetson.
+
+![](./images/TAM_screenshot_cat.png)
+
+!!! abstract "What you need"
+
+    1. One of the following Jetson:
+
+        <span class="blobDarkGreen4">Jetson AGX Orin 64GB</span>
+        <span class="blobDarkGreen5">Jetson AGX Orin (32GB)</span>
+
+    2. Running one of the following [JetPack.5x](https://developer.nvidia.com/embedded/jetpack)
+
+        <span class="blobPink1">JetPack 5.1.2 (L4T r35.4.1)</span>
+        <span class="blobPink2">JetPack 5.1.1 (L4T r35.3.1)</span>
+        <span class="blobPink3">JetPack 5.1 (L4T r35.2.1)</span>
+
+    3. Sufficient storage space (preferably with NVMe SSD).
+
+        - `6.8GB` for container image
+        - Spaces for models
+
+## Set up a container for `tam`
+
+### Clone `jetson-containers`
+
+!!! tip ""
+
+    See [`jetson-containers`' `tam` package README](https://github.com/dusty-nv/jetson-containers/tree/master/packages/vit/tam) for more infomation**
+
+```
+git clone https://github.com/dusty-nv/jetson-containers
+cd jetson-containers
+sudo apt update; sudo apt install -y python3-pip
+pip3 install -r requirements.txt
+```
+
+## How to start
+
+Use `run.sh` and `autotag` script to automatically pull or build a compatible container image.
+
+```
+cd jetson-containers
+./run.sh $(./autotag tam)
+```
+
+The container has a default run command (`CMD`) that will automatically start TAM's web server.
+
+Open your browser and access `http://<IP_ADDRESS>:12212`.
+
+## TAM web UI
+
+Check out the [official tutorial](https://github.com/gaomingqi/Track-Anything/blob/master/doc/tutorials.md) to learn how to operate the web UI.
+
+<video controls>
+<source src="./images/TAM_screencast_cat_720p-80pcnt" type="video/mp4">
+</video>
+
+## Results
+
+<video controls autoplay>
+<source src="./images/TAM_15s_1080p.mp4" type="video/mp4">
+</video>
diff --git a/docs/tutorial_text-generation.md b/docs/tutorial_text-generation.md
@@ -10,7 +10,7 @@ Interact with a local AI assistant by running a LLM with oobabooga's [`text-gene
 
         <span class="blobDarkGreen4">Jetson AGX Orin 64GB</span>
         <span class="blobDarkGreen5">Jetson AGX Orin (32GB)</span>
-        <span class="blobLightGreen4">Jetson Orin Nano Orin (8GB)</span>[^1]
+        <span class="blobLightGreen4">Jetson Orin Nano Orin (8GB)</span>⚠️[^1]
 
     2. Running one of the following [JetPack.5x](https://developer.nvidia.com/embedded/jetpack)
 
@@ -23,7 +23,7 @@ Interact with a local AI assistant by running a LLM with oobabooga's [`text-gene
         - `6.2GB` for container image
         - Spaces for models
 
-    [^1]: Lorem ipsum dolor sit amet, consectetur adipiscing elit.
+    [^1]: Limited to 7B model (4-bit quantized).
 
 ## Set up a container for `text-generation-webui`
 

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -77,7 +77,12 @@ nav:
     - Image Generation:
       - Stable Diffusion: tutorial_stable-diffusion.md
     - Vision:
+      - SAM: tutorial_sam.md
       - NanoSAM: tutorial_nanosam.md
+      - TAM: tutorial_tam.md
+    # - NanoOWL: tutorial_nanoowl.md
+    - VectorDB:
+      - nanodb: tutorial_nanodb.md
     # - Tools:
     #   - LangChain: tutorial_distillation.md
     - Tips: