An app does not react to mouse movements/clicks where AppleScript with cliclicks terminal app works

[andkirby]

tl;dr

Hammerspoon cannot simulate mouse movement or clicks in GeForceNow (or maybe other apps, and I don't know why, because of app implementation or full screen mode), where Apple Script works fine.
Why? Should I enable some special mode?.. Thank you for any feedback.
And thank you for work on this app! :)

Description

I'm trying to make some automation in a GeForceNow.
The application does not respond to mouse movements nor mouse clicks.

I've been testing this script to move a mouse through a menu in a game (feel free to add clicks if needed).
Menu items should react with hover highlighting. I tested clicks, didn't work as well.

Lua Code here

local obj = {}
obj.__index = obj

-- Metadata
obj.name = "SandboxMouse"
obj.version = "1.0"
obj.author = "Your Name <[email protected]>"
obj.homepage = "https://github.com/your-repo"
obj.license = "MIT - https://opensource.org/licenses/MIT"

-- Function to move the mouse from (x1, y1) to (x2, y2) in increments of 3 pixels
function obj:moveMouse(x1, y1, x2, y2, duration)
    local stepDistance = 1 -- Distance to move in each step
    local totalDistance = math.sqrt((x2 - x1) ^ 2 + (y2 - y1) ^ 2) -- Calculate the total distance

    -- Prevent division by zero
    if totalDistance == 0 then
        return -- No movement needed if start and end positions are the same
    end

    local steps = math.ceil(totalDistance / stepDistance) -- Calculate the number of steps needed
    local delay = (duration / steps) * 3 -- Increase the delay by a factor of 3

    -- Log for debugging
    print("Total Distance:", totalDistance)
    print("Steps:", steps)
    print("Delay per Step:", delay * 10e6)

    -- Calculate the direction vector
    local dx, dy = self:calculateDirection(x1, y1, x2, y2, totalDistance)

    -- Function to move the mouse incrementally
    local function moveStep(stepIndex)
        if stepIndex > steps then
            -- Ensure the mouse ends up exactly at (x2, y2)
            hs.mouse.absolutePosition(hs.geometry.point(x2, y2))
            return
        end

        -- Calculate the new position
        local x, y = self:calculateNewPosition(x1, y1, dx, dy, stepDistance, stepIndex)

        -- Move the mouse to the calculated position using absolutePosition
        hs.mouse.absolutePosition(hs.geometry.point(x, y))

        -- Schedule the next step
        hs.timer.doAfter(delay, function()
            moveStep(stepIndex + 1)
        end)
    end

    -- Start the movement
    moveStep(1)
end

-- Helper function to calculate the direction vector
function obj:calculateDirection(x1, y1, x2, y2, totalDistance)
    return (x2 - x1) / totalDistance, (y2 - y1) / totalDistance
end

-- Helper function to calculate the new position
function obj:calculateNewPosition(x1, y1, dx, dy, stepDistance, stepIndex)
    local x = x1 + dx * stepDistance * stepIndex
    local y = y1 + dy * stepDistance * stepIndex
    return x, y
end

-- Set up hotkey for playback: Cmd+Opt+Shift+P
hs.hotkey.bind({ "cmd", "alt", "shift" }, "T", function()
    -- Example usage: Move the mouse from (100, 100) to (500, 500) over 2 seconds
    spoon.SandboxMouse:moveMouse(305, 468, 305, 1024, 0.4)
    hs.alert.show("SandboxMouse:moveMouse() started")
end)

return obj

The app just does not respond.

Apple Script

I've decided to test other solutions, and tested Apple Script. It works via using cliclick shell application.

Apple Script Code here

-- Function to calculate the square root
on sqrt(value)
	return value ^ 0.5
end sqrt

-- Custom ceiling function
on ceiling(value)
	if value = (value as integer) then
		return value as integer
	else
		return (value as integer) + 1
	end if
end ceiling

-- Function to move the mouse from (x1, y1) to (x2, y2) over a specified duration
on moveMouse(x1, y1, x2, y2, duration)
	set stepDistance to 5 -- Increase distance to move in each step for faster movement
	set totalDistance to sqrt(((x2 - x1) ^ 2) + ((y2 - y1) ^ 2)) -- Calculate the total distance
	
	-- Prevent division by zero
	if totalDistance = 0 then
		return -- No movement needed if start and end positions are the same
	end if
	
	set steps to ceiling(totalDistance / stepDistance) -- Calculate the number of steps needed
	set delayTime to (duration / steps) * 0.5 -- Decrease the delay time for faster movement
	
	-- Calculate the direction vector
	set {dx, dy} to calculateDirection(x1, y1, x2, y2, totalDistance)
	
	-- Move the mouse incrementally
	repeat with stepIndex from 1 to steps
		-- Calculate the new position
		set {newX, newY} to calculateNewPosition(x1, y1, dx, dy, stepDistance, stepIndex)
		
		-- Convert newX and newY to strings for clarity
		set newXString to (newX as integer) as string
		set newYString to (newY as integer) as string
		
		-- Move the mouse to the calculated position using the full path to cliclick
		do shell script "/opt/homebrew/bin/cliclick m:" & newXString & "," & newYString
		
		-- Wait for the specified delay before the next step
		delay delayTime
	end repeat
	
	-- Ensure the mouse ends up exactly at (x2, y2)
	do shell script "/opt/homebrew/bin/cliclick m:" & x2 & "," & y2
end moveMouse

-- Helper function to calculate the direction vector
on calculateDirection(x1, y1, x2, y2, totalDistance)
	set dx to (x2 - x1) / totalDistance
	set dy to (y2 - y1) / totalDistance
	return {dx, dy}
end calculateDirection

-- Helper function to calculate the new position
on calculateNewPosition(x1, y1, dx, dy, stepDistance, stepIndex)
	set newX to x1 + (dx * stepDistance * stepIndex)
	set newY to y1 + (dy * stepDistance * stepIndex)
	return {newX, newY}
end calculateNewPosition

-- Example usage: Move the mouse from (305, 468) to (305, 1024) over 0.4 seconds
moveMouse(305, 468, 305, 1024, 0.4)