Add horizontal skew and vertical skew features

[guoyingtao]

https://www.hackingwithswift.com/articles/135/how-to-render-uiviews-in-3d-using-catransformlayer
https://dzone.com/articles/a-look-at-perspective-transform-correction-with-co
https://stackoverflow.com/questions/8235288/perspective-correction-of-uiimage-from-points
https://github.com/paulz/PerspectiveTransform

CATransform3D CATransform3DMakeRotation(CGFloat angle, CGFloat x, CGFloat y, CGFloat z);

[guoyingtao]

Possible steps

1. Rotate UIImageView by CATransform3DMakeRotation
2. Get corner point of rotated UIImageView
3. Use CIPerspectiveTransform to create the new image

[guoyingtao]

func CATransform3DMakePerspective(_ x: CGFloat, _ y: CGFloat) -> CATransform3D {
    var transform = CATransform3DIdentity
    transform.m34 = -1.0 / 1000.0
    transform = CATransform3DRotate(transform, y, 1, 0, 0)
    transform = CATransform3DRotate(transform, x, 0, 1, 0)
    return transform
}

[guoyingtao]

@objc func handleSliderValueChanged(_ slider: UISlider) {
    let value = CGFloat(slider.value)
    let inputImage = CIImage(image: imageView.image!)
    
    // Calculate the perspective correction values
    let inputWidth = inputImage.extent.width
    let inputHeight = inputImage.extent.height
    let topLeft = CGPoint(x: 0, y: 0)
    let topRight = CGPoint(x: inputWidth, y: 0)
    let bottomLeft = CGPoint(x: 0, y: inputHeight)
    let bottomRight = CGPoint(x: inputWidth, y: inputHeight)
    let correction = CIPerspectiveTransform(inputImage: inputImage,
                                             topLeft: topLeft,
                                             topRight: CGPoint(x: topRight.x + value * inputWidth, y: topRight.y),
                                             bottomLeft: bottomLeft,
                                             bottomRight: CGPoint(x: bottomRight.x + value * inputWidth, y: bottomRight.y))
    
    // Apply the perspective correction to the CIImage
    let outputImage = correction.outputImage
    let context = CIContext()
    let cgImage = context.createCGImage(outputImage, from: outputImage.extent)
    imageView.image = UIImage(cgImage: cgImage!)
}

In this modified function, we calculate the perspective correction values based on the value of the slider, and then apply the CIPerspectiveTransform filter to the CIImage to create the corrected output image. Finally, we create a UIImage from the corrected output image and set it as the image for the UIImageView.

Note that this implementation assumes that the image is being cropped to preserve the original aspect ratio. If you want to allow for the image to be resized to fit the screen, you will need to adjust the calculation of the topRight and bottomRight points.

[guoyingtao]

Made a little bit progress with the code below

        var transform = CATransform3DIdentity
        transform.m34 = -1.0 / 500.0
        transform = CATransform3DRotate(transform, totalRadians, 1, 0, 0)
        cropWorkbenchView.layer.transform = transform

[Karllas]

Hello,

Any plans of finishing this implementation?

[guoyingtao]

@Karllas
I haven't found a good way to build this feature yet, so I have no plans to finish it in the near future.

[rickshane]

The solution to this problem is fairly straightforward in a Core Image workflow:

There is a UX that presents two sliders to the user - a Horizontal Perspective adjustment slider with values ranging from -n to +n, centered at 0, and a Vertical Perspective adjustment slider with the same values.

Both sliders will provide input to a single function that calculates input to the Core Image CIPerspectiveTransform filter, which accepts four CIVectors that represent adjustments at the four corners of the image.

Only a single side of the image is adjusted for a given axis. A negative value for the horizontal adjustment results in the left side of the image being adjusted. A positive value for the horizontal adjustment results in the right side of the image being adjusted (see image below). As the adjustment in magnitude increases, both the x-axis and y-axis adjustment increases on the side of the adjustment (though not necessarily with equal magnitude - this is something to be tuned by the implementer). The same scenario applies for vertical adjustments.

NOTE: This depends on the design. Apple DOES adjust both sides for a given axis, but not in equal amounts and it depends on the value of the slider. The mechanics of this will need to be codified for Apple's implementation.

Regarding design of some popular tools: Photomator and Darkroom both adjust one side for a given axis. Apple adjusts both sides. Lightroom anchors in the center and pivots like a see-saw (which is similar to the demo video above).

This is a stateless solution - subsequent adjustments do not stack as transactions, they merely replace the previous value. All adjustments must be applied to the original full-sized image.

A flip transform will invert any non-zero adjustment along the same axis, i.e., if there is a negative horizontal adjustment, then flipping the image horizontally will invert the horizontal adjustment to a positive value of the same magnitude.

A 90 degree rotation will swap the horizontal and vertical slider values. It is important to still apply an inversion depending on the rotation state.

A straighten operation that adjusts the rotation degrees (such as performed by the RotationDial control) should NOT affect the slider values.

A single function taking as input the two slider values then calculates the four input values to set in the CIPerspectiveTransform filter for inputTopLeft, inputTopRight, inputBottomLeft, inputBottomRight. It is left to the implementer to determine how these calculations are generated.

NOTE: When both horizontal and vertical adjustments are made at the same time, then one of the inputs to CIPerspectiveTransform = f(horizontalAdjustment, verticalAdjustment), i.e., one of the corners adjusted will be determined by both slider values.

The most complex issue for Mantis is that of UX: how to design the user interface to accommodate the two sliders. Obviously the SlideControl may be used for both adjustments. They could be stacked in the area below the image where the rotation dial appears, or there could be a modal solution where only one control appears at a time.

[rickshane]

Here is the description of CIPerspectiveTransform from Core Image Filter Reference:

[rickshane]

[guoyingtao]

@rickshane
Thanks for the suggestion!
The challenging part is when rotating the image, how to adjust cropBox for 3d space. I haven't figured it out yet.

Update

It is not adjusting cropBox, it should be sometimes the image need to keep touching the cropBox corners while rotating. Looks like it needs more work to do for a skewed image.

RPReplay_Final1711317587.mov

[guoyingtao]

For UX part, we can directly use the design from Apple's Photos app which separated rotation with horizontal skew and vertical skew.

I have another project https://github.com/guoyingtao/Inchworm which servers the similar purpose. Once I solved this issue, I can borrow the Apple's design to Mantis.

[rickshane]

Very nice component!

[rickshane]

I have been thinking about this problem again.

It seems like the ideal solution would be to apply CATransform3DRotate to the CropWorkbenchView and not use the CIPerspectiveTransform filter. Otherwise, the image size would constantly be changing as the slider values change. You would then use CIPerspectiveTransform when you go to export the image since I don't think you can apply CATransform3DRotate to an image.

But then given the same input for the x, y slider values, how would you visually match the output of the CropWorkbenchView transformation and the output of CIPerspectiveTransform filter when exporting the image?

You would need the four corner values in the image to pass to CIPerspectiveTransform. Can you get this by a convert() call from CropAuxiliaryIndicatorView coordinates to the ImageContainerView coordinates?

[guoyingtao]

@rickshane
I agree with you and that is also a possible solution what I think too.

There will be some math calculation work when doing image crop but the UI interaction part also need more work. I made a little bit progress to mimic the skew operation without any rotation (Which may need the image keeps touching the cropBox corners while rotating in some scenarios)

[rickshane]

Is this code in a branch or fork?

[guoyingtao]

No, it's just some test code and far from being usable

[rickshane]

When you rotate an image, are you changing the zoomScale and position of WorkbenchView to line up an edge (or edges) of a rectangle to the cropBox? Where is that current logic in the code?

[rickshane]

I think this article might provide the solution to the skew step:

https://stackoverflow.com/questions/9470493/transforming-a-rectangle-image-into-a-quadrilateral-using-a-catransform3d/18606029#18606029

It provides a Swift class that can compute a CATransform3D that converts a rectangle to a quadrilateral based on the 4 corners of the quadrilateral. So this means that the two inputs from the sliders (horizontal and vertical) would be used to compute the four points of the transformed view (or the adjusted image). You could then pass those 4 inputs to the logic in the above article to transform the CropWorkbenchView and also use the same 4 inputs to pass to CIPerspectiveTransform during the crop step.

This solution is NOT using CATransform3DRotate.

[guoyingtao]

When you rotate an image, are you changing the zoomScale and position of WorkbenchView to line up an edge (or edges) of a rectangle to the cropBox? Where is that current logic in the code?

Yes, but it is for the WorkbenchView without 3d transform, the logic is in the CropView.adjustWorkbenchView function.

private func adjustWorkbenchView(by radians: CGFloat) {
        let width = abs(cos(radians)) * cropAuxiliaryIndicatorView.frame.width + abs(sin(radians)) * cropAuxiliaryIndicatorView.frame.height
        let height = abs(sin(radians)) * cropAuxiliaryIndicatorView.frame.width + abs(cos(radians)) * cropAuxiliaryIndicatorView.frame.height
        
        cropWorkbenchView.updateLayout(byNewSize: CGSize(width: width, height: height))
        
        if !isManuallyZoomed || cropWorkbenchView.shouldScale() {
            cropWorkbenchView.zoomScaleToBound(animated: false)
            isManuallyZoomed = false
        } else {
            cropWorkbenchView.updateMinZoomScale()
        }
        
        cropWorkbenchView.updateContentOffset()
    }

[rickshane]

I created a topic branch that contains some basic logic (UX and state management) to support three rotation types (straighten, vertical skew, horizontal skew) in the EmbeddedViewController. Selecting Straighten works as expected, Selecting horizontal or vertical skew does nothing (other than setting some state). Note: I do not maintain rotation degrees state for each type yet.

Here is a video:

Screen.Recording.2024-05-13.at.10.08.19.AM.mov

I will issue a pull request in case you want to use some of this code.

[rickshane]

When you rotate an image, are you changing the zoomScale and position of WorkbenchView to line up an edge (or edges) of a rectangle to the cropBox? Where is that current logic in the code?

Yes, but it is for the WorkbenchView without 3d transform, the logic is in the CropView.adjustWorkbenchView function.

private func adjustWorkbenchView(by radians: CGFloat) {
        let width = abs(cos(radians)) * cropAuxiliaryIndicatorView.frame.width + abs(sin(radians)) * cropAuxiliaryIndicatorView.frame.height
        let height = abs(sin(radians)) * cropAuxiliaryIndicatorView.frame.width + abs(cos(radians)) * cropAuxiliaryIndicatorView.frame.height
        
        cropWorkbenchView.updateLayout(byNewSize: CGSize(width: width, height: height))
        
        if !isManuallyZoomed || cropWorkbenchView.shouldScale() {
            cropWorkbenchView.zoomScaleToBound(animated: false)
            isManuallyZoomed = false
        } else {
            cropWorkbenchView.updateMinZoomScale()
        }
        
        cropWorkbenchView.updateContentOffset()
    }

Is this a correct statement to the problem?:

You have currently solved how to align a rotated rectangle, with known centerPoint (the cropWorkBenchView.frame) along a bounding rectangle (the CropBox). Now you need to solve how to align a rotated Isosceles trapezoid, with known centerPoint (which is just a rectangle with two right triangles on each side, with known vertex lengths and angles) along a bounding rectangle (the CropBox).

Is that the problem that needs to be solved?

[guoyingtao]

Looks like it is.
The 3d interaction demo I was showing includes only skew + special translation. When adding scale, rotation and random translation, we still need to keep the CropBox inside the image, and sometimes the images edges need to touch the CropBox vertices while rotating.
I haven't figured it out a good solution and haven't got enough time to work on it.

[rickshane]

Does your 3d interaction demo use "CATransform3DRotate" ?

[guoyingtao]

It uses CATransform3DRotate + CATransform3DTranslate (When rotation angle is greater than a specified angle, it begins to move the rotation axis with CATransform3DTranslate to mimic the similar Photo.app's skew interactions on iPhone.)

[rickshane]

Does the translate start getting called when rotation >= (+/- 10 degrees)?

[rickshane]

I updated the PR to add state management that updates RotationDial value when changing modes between straighten, horizontal skew, and vertical skew.

I also added basic horizontal and vertical skewing by applying a transform to ImageContainer instead of CropWorkbenchView (applying transform to CropWorkbenchView gave unexpected results). For now, I only skew one side of image depending on whether slider value is positive or negative. One bug I cannot figure out is that the opposite side of the skew is also skewing. This is strange, because I have a separate test harness that does not use Mantis code and the non-skewed side stays at original anchor points.

This skew method does not use CATransform3D rotate or translate. I will try to add some code to skew opposite side when degrees >= threshold (+/- 10 degrees?) to look more like Apple Photos app.

I notice that the contentOffset is in the wrong place if I skew image and then try to straighten the image (rotate). So cannot successfully switch between rotating image and skewing at this point.

Here is a video:

Screen.Recording.2024-05-14.at.2.22.46.PM.mov

[rickshane]

I just updated the PR to apply the CATransform3D to the CropWorkbenchView.layer instead of the ImageContainer.layer. I just needed to normalize the coordinates against the cropBoxFrame.

So CropWorkbenchView now transforms as expected. But the ImageContainer is not correct. Perhaps the transform must be applied to ImageContainer.layer.transform, as well.

I made the CropWorkbenchView.background color = red to show the disparity between CropWorkbenchView and ImageContainer.

Here is a video:

Screen.Recording.2024-05-15.at.1.56.05.PM.mov

[rickshane]

The problem between skew and straighten is probably two separate issues, and depends on the order of operations.

When going from straighten operation to skew: this will require correctly concatenating the transforms.

Going from skew to straighten operation will require the same concatenation, but then an adjustment (which is related to what you were trying to solve in adjustWorkbenchView() method).

So I will focus on getting the concatenation to work going from straighten to skew.

[rickshane]

I just updated the PR to apply the skew transform to the ImageContainer after applying it to the CropWorkbenchView.

It looks like this is the correct technique. It works perfectly in my other test harness. However in Mantis, it is not perfectly lined up. There is still some offset bug I need to figure out.

Here is a video:

Screen.Recording.2024-05-15.at.6.16.02.PM.mov

[rickshane]

Looks like it is. The 3d interaction demo I was showing includes only skew + special translation. When adding scale, rotation and random translation, we still need to keep the CropBox inside the image, and sometimes the images edges need to touch the CropBox vertices while rotating. I haven't figured it out a good solution and haven't got enough time to work on it.

Would you please provide a video example from Photos app where rotating a skewed image is touching the CropBox and CropBox stays inside the image?

Thanks

[guoyingtao]

@rickshane
Sure. You can see the example below. Also thanks for working on this issue.

Simulator.Screen.Recording.-.iPhone.15.Plus.-.2024-05-15.at.14.29.23.mp4

[rickshane]

Fixed imageView alignment issue when skewing image. Commenting out the call to set imageView.contentMode to .scaleAspectFit seems to fix this bug.

Video:

Screen.Recording.2024-05-16.at.12.04.04.PM.mov

[rickshane]

@guoyingtao

I just noticed that Mantis rotates image in opposite direction to Apple Photos app. Is this by design?

In fact, most editing apps rotate image in opposite direction to Apple Photos app.

[guoyingtao]

@guoyingtao

I just noticed that Mantis rotates image in opposite direction to Apple Photos app. Is this by design?

In fact, most editing apps rotate image in opposite direction to Apple Photos app.

I didn't notice it before, but I feel like the rotating direction is more natural in Mantis than Apple Photo app.
When you move rotation dial, you will feel that the photo rotates to the same direction with your finger which I think is more natural.

[rickshane]

@guoyingtao

It looks like this function in CropWorkbenchView is never called. May it be removed?

func shouldScale() -> Bool {
return contentSize.width / bounds.width <= 1.0
|| contentSize.height / bounds.height <= 1.0
}

[guoyingtao]

It is used in the function adjustWorkbenchView in CropView

[rickshane]

@guoyingtao

My mistake. Apologies.

I am still trying to understand the math for resizing the cropWorkbenchView upon rotation to align the auxiliaryIndicatorView. The code in the first two lines of adjustWorkbenchView() is almost similar to the formula for rotating a view using a matrix affine transform except one operator is positive instead of negative.

Please explain the first two lines in adjustWorkbenchView():

let width = abs(cos(radians)) * cropAuxiliaryIndicatorView.frame.width +
abs (sin(radians)) * cropAuxiliaryIndicatorView.frame.height
let height = abs(sin(radians)) * cropAuxiliaryIndicatorView.frame.width +
abs (cos (radians)) * cropAuxiliaryIndicatorView.frame.height

How does this work? Is this a standard formula? Is there a reference somewhere explaining it?

A drawing would be the most helpful.

[guoyingtao]

@rickshane
I did the math and no reference for it.
You can check the drawing below for the basic ideas.

[rickshane]

Thanks for the diagram.

The math made sense once I hid the ImageContainer and saw the bounds change of the CropWorkbenchView.

[rickshane]

@guoyingtao

I just updated the PR with added support for concatenating rotation and skew operations.

I have disabled the flip and adjustCropWorkbenchView calls until those are solved.

Here is a video update:

Screen.Recording.2024-05-25.at.3.47.13.PM.mov

[Karllas]

@rickshane @guoyingtao is this PR planned to be finished anytime soon?

[guoyingtao]

@Karllas
The UI interactions is complicated and we haven't figured it out a good solution yet. So this feature won't be finished anytime soon.