Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/OthersideAI/self-operating-computer
/ functions
Functions
55 in github.com/OthersideAI/self-operating-computer
⨍
Functions
55
◇
Types & classes
3
↓ 8 callers
Function
capture_screen_with_cursor
(file_path)
operate/utils/screenshot.py:11
↓ 8 callers
Function
clean_json
(content)
operate/models/apis.py:1008
↓ 7 callers
Function
get_user_first_message_prompt
()
operate/models/prompts.py:265
↓ 7 callers
Function
get_user_prompt
()
operate/models/prompts.py:260
↓ 7 callers
Function
supports_ansi
Check if the terminal supports ANSI escape codes
operate/utils/style.py:19
↓ 5 callers
Function
call_gpt_4o
(messages)
operate/models/apis.py:65
↓ 5 callers
Function
confirm_system_prompt
On `Exception` we default to `call_gpt_4_vision_preview` so we have this function to reassign system prompt in case of a previous failure
operate/models/apis.py:984
↓ 4 callers
Function
get_system_prompt
Format the vision prompt more efficiently and print the name of the prompt used
operate/models/prompts.py:210
↓ 4 callers
Function
get_text_coordinates
Gets the coordinates of the text element at the specified index as a percentage of screen width and height. Args: result (list): The
operate/utils/ocr.py:66
↓ 4 callers
Function
get_text_element
Searches for a text element in the OCR results and returns its index. Also draws bounding boxes on the image. Args: result (list): Th
operate/utils/ocr.py:10
↓ 4 callers
Function
gpt_4_fallback
(messages, objective, model)
operate/models/apis.py:968
↓ 4 callers
Method
initialize_openai
(self)
operate/config.py:50
↓ 4 callers
Method
require_api_key
(self, key_name, key_description, is_required)
operate/config.py:152
↓ 2 callers
Function
convert_percent_to_decimal
(percent)
operate/utils/misc.py:5
↓ 2 callers
Method
write
(self, content)
operate/utils/operating_system.py:10
↓ 1 callers
Function
add_labels
(base64_data, yolo_model)
operate/utils/label.py:55
↓ 1 callers
Function
call_claude_3_with_ocr
(messages, objective, model)
operate/models/apis.py:759
↓ 1 callers
Function
call_gemini_pro_vision
Get the next action for Self-Operating Computer using Gemini Pro Vision
operate/models/apis.py:259
↓ 1 callers
Function
call_gpt_4o_labeled
(messages, objective, model)
operate/models/apis.py:537
↓ 1 callers
Function
call_gpt_4o_with_ocr
(messages, objective, model)
operate/models/apis.py:311
↓ 1 callers
Function
call_o1_with_ocr
(messages, objective, model)
operate/models/apis.py:424
↓ 1 callers
Function
call_ollama_llava
(messages)
operate/models/apis.py:681
↓ 1 callers
Function
call_qwen_vl_with_ocr
(messages, objective, model)
operate/models/apis.py:142
↓ 1 callers
Method
click_at_percentage
( self, x_percentage, y_percentage, duration=0.2, circle_radius=50,
operate/utils/operating_system.py:39
↓ 1 callers
Function
compress_screenshot
(raw_screenshot_filename, screenshot_filename)
operate/utils/screenshot.py:30
↓ 1 callers
Function
evaluate_final_screenshot
Load the final screenshot and return True or False if it meets the given guideline.
evaluate.py:88
↓ 1 callers
Function
format_evaluation_prompt
(guideline)
evaluate.py:69
↓ 1 callers
Function
get_click_position_in_percent
Calculates the click position at the center of the bounding box and converts it to percentages. :param coordinates: A tuple of the bounding
operate/utils/label.py:141
↓ 1 callers
Function
get_label_coordinates
Retrieves the coordinates for a given label. :param label: The label to find coordinates for (e.g., "~1"). :param label_coordinates: Dic
operate/utils/label.py:29
↓ 1 callers
Function
get_next_action
(model, messages, objective, session_id)
operate/models/apis.py:34
↓ 1 callers
Function
get_test_model
()
evaluate.py:136
↓ 1 callers
Method
initialize_anthropic
(self)
operate/config.py:124
↓ 1 callers
Method
initialize_google
(self)
operate/config.py:95
↓ 1 callers
Method
initialize_ollama
(self)
operate/config.py:111
↓ 1 callers
Method
initialize_qwen
(self)
operate/config.py:72
↓ 1 callers
Function
is_overlapping
(box1, box2)
operate/utils/label.py:40
↓ 1 callers
Function
main
()
evaluate.py:152
↓ 1 callers
Function
main
Main function for the Self-Operating Computer. Parameters: - model: The model used for generating responses. - terminal_prompt: A st
operate/operate.py:33
↓ 1 callers
Function
main_entry
()
operate/main.py:9
↓ 1 callers
Method
mouse
(self, click_detail)
operate/utils/operating_system.py:28
↓ 1 callers
Function
operate
(operations, model)
operate/operate.py:134
↓ 1 callers
Function
parse_eval_content
(content)
evaluate.py:74
↓ 1 callers
Method
press
(self, keys)
operate/utils/operating_system.py:18
↓ 1 callers
Method
prompt_and_save_api_key
(self, key_name, key_description)
operate/config.py:162
↓ 1 callers
Function
run_test_case
Returns True if the result of the test with the given prompt meets the given guideline for the given model.
evaluate.py:119
↓ 1 callers
Method
save_api_key_to_env
(key_name, key_value)
operate/config.py:184
↓ 1 callers
Function
supports_ansi
Check if the terminal supports ANSI escape codes
evaluate.py:32
↓ 1 callers
Method
validation
Validate the input parameters for the dialog operation.
operate/config.py:131
Method
__init__
(self)
operate/config.py:31
Method
__init__
(self, model, message="Model not recognized")
operate/exceptions.py:9
Method
__new__
(cls)
operate/config.py:25
Method
__str__
(self)
operate/exceptions.py:14
Function
get_last_assistant_message
Retrieve the last message from the assistant in the messages array. If the last assistant message is the first message in the array, return N
operate/models/apis.py:954
Function
parse_operations
(response)
operate/utils/misc.py:17
Function
validate_and_extract_image_data
(data)
operate/utils/label.py:10