Question about the AI2-THOR build.

[liuxz-cs]

Excuse me, I want to know if we can define the actions and scenes (for example the shape of agents in multi-agent tasks) by rewriting the AI2THOR-build file. And if we can, how can we write the file to define our own tasks? Do you have some documentations or guildlines that we can learn to modify the AI2THOR-build file for our own tasks?

[mattdeitke]

This will definetly be more challenging than simply using the Python API. Often, if you can simply define the task as a sequence of subtasks already defined, that would be much easier. But, if you're feeling adventurous enough, we use Unity to build the backend environment, which is then accessible from a Python API.

---

Backend Setup.

1. Installing the Unity editor (current version in use is 2019.4.2 LTS, but this is subject to change and will be updated on the README.md).
2. Clone ai2thor locally
3. Within the ai2thor folder, open up ai2thor/unity within the Unity editor. This will take about 30 minutes to 2 hours to first open up, since it has to load in all the assets, but after that re-opening up the unity folder should be quick.
4. Within the Unity project folder, then search for a scene (e.g., type 'FloorPlan1'), and click on FloorPlan1_physics.unity to open it up within the editor.

You should then see a window that looks something like the following, which will allow you to interactively edit the scenes:

---

Building locally. This allows you to play with your changes from the Python API.

1. Install invoke (e.g., pip install invoke)
2. Run invoke local-build from the cloned ai2thor repo. This may take about 10-15 minutes by default. To speed it up, or test specific scenes, check out ai2thor/unity/Assets/Editor/Build.cs and change the list of scenes to only be a scene or two (e.g., FloorPlan1_physics.unity and FloorPlan1_15.unity).
3. Start AI2-THOR within Python and run

from ai2thor.controller import Controller
controller = Controller(
    local_executable_path="<BASE_DIR>/unity/builds/thor-OSXIntel64-local/thor-OSXIntel64-local.app/Contents/MacOS/AI2-Thor"
)

where base DIR is the path to the cloned ai2thor folder. All updates should then be accessible.

---

Adding actions. The agent you are targeting (e.g., there is a Drone agent, a Big Blue default agent, and a LoCoBot agent), will change where you'll want to add actions.

• All agents inherit actions from the BaseFPSAgentController, and the default agent has actions updated in the PhysicsRemoteFPSAgentController while the drone has actions update in the DroneFPSAgentController script.

It's generally safe to just copy the style of existing actions and use them as a template. But, actions that say ServerAction as a parameter are being phased out for actions that do not. The latter should be easier to work with. Check out this Teleport action for a recent example. The syntax to add actions is based on C# and Unity, so for that part I'd look at their documentations.

---

Adding scenes and objects. Probably easiest to simply duplicate an existing scene within Assets/Scenes and use it as a template. If you name it MyScene.unity from the Assets/Scenes directory, it'll be available on the local build with

from ai2thor.controller import Controller
controller = Controller(scene='MyScene')
controller.reset('MyScene')

There's a bunch of extra components added to scenes and objects that make them work, whereas starting a new scene directly would not. Similarly for objects, just duplicate an existing object and possibly tinker around with its properties and parameters so that it's bounding box fits the size of the new 3D object, its object type is reasonable, and its object name is reasonable.

[liuxz-cs]

This will definetly be more challenging than simply using the Python API. Often, if you can simply define the task as a sequence of subtasks already defined, that would be much easier. But, if you're feeling adventurous enough, we use Unity to build the backend environment, which is then accessible from a Python API.

Backend Setup.

1. Installing the Unity editor (current version in use is 2019.4.2 LTS, but this is subject to change and will be updated on the README.md).
2. Clone ai2thor locally
3. Within the ai2thor folder, open up ai2thor/unity within the Unity editor. This will take about 30 minutes to 2 hours to first open up, since it has to load in all the assets, but after that re-opening up the unity folder should be quick.
4. Within the Unity project folder, then search for a scene (e.g., type 'FloorPlan1'), and click on FloorPlan1_physics.unity to open it up within the editor.

You should then see a window that looks something like the following, which will allow you to interactively edit the scenes:

Building locally. This allows you to play with your changes from the Python API.

1. Install invoke (e.g., pip install invoke)
2. Run invoke local-build from the cloned ai2thor repo. This may take about 10-15 minutes by default. To speed it up, or test specific scenes, check out ai2thor/unity/Assets/Editor/Build.cs and change the list of scenes to only be a scene or two (e.g., FloorPlan1_physics.unity and FloorPlan1_15.unity).
3. Start AI2-THOR within Python and run

from ai2thor.controller import Controller
controller = Controller(
    local_executable_path="<BASE_DIR>/unity/builds/thor-OSXIntel64-local/thor-OSXIntel64-local.app/Contents/MacOS/AI2-Thor"
)

where base DIR is the path to the cloned ai2thor folder. All updates should then be accessible.

Adding actions. The agent you are targeting (e.g., there is a Drone agent, a Big Blue default agent, and a LoCoBot agent), will change where you'll want to add actions.

• All agents inherit actions from the BaseFPSAgentController, and the default agent has actions updated in the PhysicsRemoteFPSAgentController while the drone has actions update in the DroneFPSAgentController script.

It's generally safe to just copy the style of existing actions and use them as a template. But, actions that say ServerAction as a parameter are being phased out for actions that do not. The latter should be easier to work with. Check out this Teleport action for a recent example. The syntax to add actions is based on C# and Unity, so for that part I'd look at their documentations.

Adding scenes and objects. Probably easiest to simply duplicate an existing scene within Assets/Scenes and use it as a template. If you name it MyScene.unity from the Assets/Scenes directory, it'll be available on the local build with

from ai2thor.controller import Controller
controller = Controller(scene='MyScene')
controller.reset('MyScene')

There's a bunch of extra components added to scenes and objects that make them work, whereas starting a new scene directly would not. Similarly for objects, just duplicate an existing object and possibly tinker around with its properties and parameters so that it's bounding box fits the size of the new 3D object, its object type is reasonable, and its object name is reasonable.

Okay, I will try it. Thanks a lot.

[sarthakchittawar]

On executing invoke local-build I get the following error:

It seems that the _build function is not executing, although after debugging I saw that it returned an exit status 0.
Moreover, I feel like the invoke local-build call is not functioning correctly, as I had to manually install all dependencies, instead of it being part of the process.
Even if I create the build folder manually, the .app file is not generated. Is there a solution to this?

[zhiyuan-gao]

On executing invoke local-build I get the following error: It seems that the _build function is not executing, although after debugging I saw that it returned an exit status 0. Moreover, I feel like the invoke local-build call is not functioning correctly, as I had to manually install all dependencies, instead of it being part of the process. Even if I create the build folder manually, the .app file is not generated. Is there a solution to this?

Hℹ, I also encountered this problem, did you find any solution?