Calculate position,rotation, scale, offsets, then manipulate image (javascript)
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I have a opencv script that finds specific descriptors on an image. they are just coordinate points. I am able to take those points and find similar/same images in a corpus. In some cases the image I am given will match according to openCV based on the descriptor points. I need need to be able to take the coordinates of the "corpus" image and then figure out how to manipulate the given image so that its properly scaled, rotated and padded to match the corpus image.

for context here is an example - corpus image on the right, given images on the left.
https://www.dropbox.com/s/y277szleov5co3j/2020-10-20_13-31-44.png?dl=0

for each image you will be given its image size and the x,y coordinates of each point. Open CV will tell us which point is equal to the corresponding point between the images as seen here

https://www.dropbox.com/s/j9uz85l9n6cngrz/2020-10-20_13-28-14.png?dl=0

so you can assume in this more simplistic view that we will give you coordinate points for each image in order. coordinate 1 is assumed to match coordinate 1 on the corpus on so on...

we need you to compute how to scale, rotate and pad the given image to match how it would be in the corpus image. In the case of a cropped image we would need to "fill" the empty space from the offset with a white background so the final image output looks like its stuck on the same area of the corpus image.

-here is the bounty. all can be computed in browser (fine to assume chrome)

-upload corpus image with 5 coordinates (assume order matters)
-upload "given" image with 5 coordinates (assume order matters)

you will output the image manipulated and on top of the corpus image with the following data

-rotation - in degrees

-scale - as a percent

-offset location of each of the 4 corners of the image (since it can be rotated we need all 4 points)

a visual of the final output would be -

https://www.dropbox.com/s/tcpkikz5i6op61k/2020-10-20_13-47-16.png?dl=0

notice the overlaid image is in the right place, scaled and rotated. then i have filled the remaining empty space with white (was trying to semi transparent so you can see the alignment with the corpus background)

you can use any libs you want. this is more a POC so im looking to see the actual algos and if we can reasonably get this working

Winner of this will get 200, 100 via bounty and 100 via bonus

Can you throw some data to the mix?
kostasx 1 month ago
Is the scaling an homothety ? the scale would then be the same in X and Y
kerncy 1 month ago
hey Kerncy, great question. lets assume yes for the purpose of this POC Kostasx, i can post some sample data shortly. I was fearful it wouldnt be mathematically perfect but let me see what i can cook up for us
Qdev 1 month ago
awarded to kerncy
Tags
javascript

Crowdsource coding tasks.

1 Solution


Hi QDev,

Here is a working demo using your fox image :
https://web.kern-info.eu/imageTransform/bounty.html

When you click on the image, the transformation will be computed.

I have estimated the points position in your image using GIMP and setting them as coordinates in the two textareas. They contains array of points in the same order (1, 2, 3, 4, 5).

All the code you need is in the source page. Regarding the algorithm :

  • first compute the mean point of all the points (pointA1, pointA2, pointA3, ... ) would give you mean A. (pointB1, pointB2, pointB3, ...) would give you meanB. The delta between the coordinates of those means would be your translation.
  • scaling is computed using the length scaling between each mean and the first point of each image (meanA-pointA) / (meanB-pointB)
  • rotation is computed using the angle between the vectors (meanA pointA1), (meanB pointB1)

I then set the css transform to see the matching.

Do not hesitate if you have any question.

PS : superposition is not exact because of estimation of the points positions, but also the too big scaling difference that will increase computational errors

Geez man this is great! I’ll work on a more precise set of points. Based on the quick trial it seems to be perfect. I need to dig more tomorrow. Do you have a rough or easy way to approx the computation time?
Qdev 1 month ago
Computation time is linear, as it just uses a small part of the points set to make the computation. Opérators are only standard math fonction (+ - * / cos sqrt) and no loop is used (only for mean computation but it can bé removed). In fact it will also work even if you set only two points on each image. It is enough to get the transformation in two dimensions.
kerncy 1 month ago
hey there i tried to move this and run some more specific point coordinates - its directionally working but the alignment didnt go as expected.. I cant tell if my coorindate is wrong somewhere or a calc issue, ideas? https://repl.it/@quotient1/reunsquish#index.html
Qdev 1 month ago
Hi QDev, it seems to be more a computational issue, I will have to check it tonight because I don't have access to my code for the moment. I will keep you up to date.
kerncy 1 month ago
thanks man, i did a double check on points to make sure they are right and they seem to be... really appreciate the help on this
Qdev 1 month ago
OK I found the bug, it's on line 87, please update javascript with the following line : image.setAttribute("style", "transform-origin : " + meanB[0] + "px " + meanB[1] + "px; transform: translate(" + motion[0] + "px, " + motion[1] + "px) rotate(" + angle + "deg) scale(" + scaling + ")"); I was missing the transform-origin part that modifies the rotation. Rotation was done around 0,0 instead of meanX,meanY on the image. This causes the failure of position. I made a fork on repl.it but not sure the link works : https://repl.it/repls/IllustriousKindAutosketch#index.html
kerncy 1 month ago
OK I found the bug, it's on line 87, please update javascript with the following line : image.setAttribute("style", "transform-origin : " + meanB[0] + "px " + meanB[1] + "px; transform: translate(" + motion[0] + "px, " + motion[1] + "px) rotate(" + angle + "deg) scale(" + scaling + ")"); I was missing the transform-origin part that modifies the rotation. Rotation was done around 0,0 instead of meanX,meanY on the image. This causes the failure of position. I made a fork on repl.it but not sure the link works : https://repl.it/repls/IllustriousKindAutosketch#index.html EDIT : I updated my code too, result is now better new solution : https://web.kern-info.eu/imageTransform/bounty.html old solution : https://web.kern-info.eu/imageTransform/bounty2.html
kerncy 1 month ago
bug fix was perfect! is there anyway with canvas for me to download the transformed image with a white background? so its the same shape and size of the corpus but white fills the exterior empty regions?
Qdev 1 month ago
Ok i see your needs. I will work on a solution where you will be able to download generated image
kerncy 1 month ago
totally ideal if its png or bmp so it doesnt introduce compression artifacts... thx man
Qdev 1 month ago
Hi I made the code update : https://web.kern-info.eu/imageTransform/bounty.html You can now set the image url and the data field. You will then have to click on "load images" to see your images displayed in the browser. If you then click on "compute & download", it will compute the transformation, display the animation, and finally download the transformed image as "filename.png".
kerncy 1 month ago
Thank you QDev for the reward and the tip ! I see you still have an other bounty that has not been rewarded yet : https://bountify.co/efficient-pattern-matching-algo-python-or-js maybe you can have a look ?
kerncy 1 month ago