EQA merge #254
EQA merge #254
Conversation
…/stretch_ai into hello-peiqi/grapheqa_merge
|
|
||
|
|
||
| class SparseVoxelMap(SparseVoxelMapBase): | ||
| def __init__( |
There was a problem hiding this comment.
It would be good to add a docstring here, this is a lot of params (mostly borrowed - can use @OVERRIDES)
| DYNAMEM_VISUAL_GROUNDING_PROMPT, model="gpt-4o-2024-05-13" | ||
| ) | ||
|
|
||
| def find_alignment_over_model(self, queries: str): |
| print("Points close the the point are not similar to the text!") | ||
| return torch.max(alignments[distances < distance_threshold]) >= similarity_threshold | ||
|
|
||
| def get_2d_map( |
There was a problem hiding this comment.
Is this different from the base one? If not - just override
if so maybe we could do some refactoring for less duplication?
| if debug: | ||
| import matplotlib.pyplot as plt | ||
|
|
||
| plt.subplot(2, 2, 1) |
There was a problem hiding this comment.
you can probably delete this debug code
| print(f"Error: {e}") | ||
| return None | ||
|
|
||
| def localize_with_mllm(self, text: str, debug=True, return_debug=False): |
There was a problem hiding this comment.
this new function could probably use some documentation
src/stretch/visualization/rerun.py
Outdated
| @@ -220,6 +221,9 @@ def __init__( | |||
| logger.warning("Docker environment detected. Disabling GUI.") | |||
| rr.init("Stretch_robot", spawn=spawn_gui) | |||
|
|
|||
| if output_path is not None: | |||
| rr.save(output_path / "rerun_log.rrd") | |||
There was a problem hiding this comment.
do you find these rerun logs useful?
There was a problem hiding this comment.
Not yet, but I guess it is for automatically saving rerun res; since grapheqa does that, I think it is useful especially for some researchers want to collect demo for their websites.
scripts/run_stretch_ai_gpu_client.sh
Outdated
| @@ -95,7 +95,6 @@ fi | |||
| # hellorobotinc/stretch-ai_cuda-11.8:$VERSION: the docker image to run | |||
| run_docker_command run \ | |||
| -it \ | |||
| --runtime nvidia \ | |||
There was a problem hiding this comment.
why do you get rid of this? is it no longer necessary?
There was a problem hiding this comment.
It gives me bugs
There was a problem hiding this comment.
Can you still run with access to the NVIDIA hardware?
projects/grapheqa/robot_agent.py
Outdated
| self._manipulation_radius = parameters["motion_planner"]["goals"]["manipulation_radius"] | ||
| self._voxel_size = parameters["voxel_size"] | ||
|
|
||
| # self.image_processor = VoxelMapImageProcessor( |
projects/grapheqa/robot_agent.py
Outdated
|
|
||
| def execute_action( | ||
| self, | ||
| text: str, |
There was a problem hiding this comment.
documentation - say what actions can be?
| Describe the CURRENT IMAGE. Pay special attention to features that can help answer the question or select future actions. | ||
| Describe the SCENE GRAPH. Pay special attention to features that can help answer the question or select future actions. | ||
| """ | ||
| prompt += "You should go near the blue couch before answering the question with confidence. You should see a full image of the couch before answering with confidence" |
There was a problem hiding this comment.
This is a remnant from initial debugging of the planner and can be removed.
* update qwen * update qwen25client * initial commit * checkpoint * bug fixing * bug fixing * precommit check * debug exploration * fix qwen bug * update * find images update * checkpoint * small fix * update history * update * update model * add siglip2 * LFD update (#259) * update lfd docs * fix bug * change hyperparameter * update docker for data collection * add docker file * update docs to add docker instructions --------- Co-authored-by: Chris Paxton <[email protected]> * update exploration * fix exploration bugs * change explrotion target * save changes * fix navigation bug * discord * fix bugs * reproduce fix bugs commit * update back * rerun save --------- Co-authored-by: Mohamed Fazil <[email protected]> Co-authored-by: Peiqi Liu <[email protected]> Co-authored-by: Chris Paxton <[email protected]>
…-robot/stretch_ai into hello-peiqi/grapheqa_merge
* update Openai * add test image * update qwen
Anyone want to test this system based on this instruction Codes are currently on hello-peiqi/grapheqa_merge branch.
You don't have to worry about discord bot.
I also modified DynaMem and AI Pick up. So please also check:
Start with some simple questions such as "is there a monitor on the table?" "what is the color of the chair"