+ Reply to Thread
Results 1 to 15 of 15

Moving all data points closer to the trend line on a graph

  1. #1
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Post Moving all data points closer to the trend line on a graph

    Hi there

    I have conducted an experiment for which I have a data set. The graph in both axes contain degree values (angles). The experiment was about comparing the steer (Yaw) and lean (Roll) angles incurred while riding a bicycle.

    The trend line for the data points is not generating a good R^2 (correlation coefficient value). I know that there will have been errors involved and so I want to know if its possible to be able to move all the data points on the graph (say by 10% or 1°) closer to the trend line in both the x and y axes. This will entail all the data points moving in each axis differently, as each will be at a different location relative to the trend line.

    Is such a thing possible? It would save me so much time. I have attached my worksheet with the data set and graph for your kind reference. I am new to this forum and not entirely sure if my worksheet is attached or not. I have added it under the "Manage Attachments" option below. Please let me know if it is there.

    Many thanks in advance
    Tarun
    Attached Files Attached Files

  2. #2
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,827

    Re: Moving all data points closer to the trend line on a graph

    Short but useless answer -- yes such a thing is possible.

    How much of the question is specifically Excel, and how much is broader statistics/math/geometry? We are usually pretty good with Excel programming questions, but we don't always have the statistics/match/geometry expertise that I find is so important before you get the programming question. If you can help us understand the algorithm you want to use, we should be able to help you program that into Excel.

    If this is more of a statistics/math/geometry question, I would wonder exactly what you want to do. A statistical purist might wonder if it is even appropriate to "massage" the data just to make the R^2 value look better. To such a purist, the only correct way to improve R^2 is to examine the experimental protocol and figure out how to reduce the experimental error.

    When you talk about moving a data point 10% closer, what exactly do you have in mind? I could visualize some variation of "compute the perpendicular distance between data point and trendline, then compute 90% of that distance, then compute a new point along the same perpendicular but 10% closer." Is that the kind of geometry problem you envision for this, or do you have a different way of visualizing distance and closer? I expect that most of the work for this kind of algorithm is finding formulas for "perpendicular distance" and "finding a point that is a certain perpendicular distance from a line". I expect a geometry text or tutorial would have such formulas.

    Help us understand exactly what you are trying to do, and we should be able to help you program that into Excel.
    Quote Originally Posted by shg
    Mathematics is the native language of the natural world. Just trying to become literate.

  3. #3
    Forum Expert Crooza's Avatar
    Join Date
    10-19-2013
    Location
    Hunter Valley, Australia
    MS-Off Ver
    Excel 2003 /7/10
    Posts
    2,082

    Re: Moving all data points closer to the trend line on a graph

    Here's How I'd do it. It follows the logic of Mr Shorty essentially too in that it takes the perpendicular distance to the line and moves the point a percentage closer.

    If you change D4 you'll see the orange dots move closer.

    0% has them on the same point as the blue dots, -100% has them on the line of best fit and -10% for example moves them 10% closer form the blue dot point of origin. If you get large variances in the x and y scales on the chart it may look like they aren't moving perpendicular but I think that is a function of the skewed scales
    Happy with my advice? Click on the * reputation button below

  4. #4
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    Hi Crooza

    Thank you so much for your effort in this. Yes this is exactly what I wanted. I hadn't realized before that the perpendicular distance from each point to the trend line is the shortest distance. Just curious, how were you able to figure out the algorithm to this? Did you refer to any book material etc.? I am an excel newbie so it would be beyond me to have worked it out in the short time span I have to do this!

    Cheers

  5. #5
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    Hi MrShorty

    Thank you very much for your help and for homing into the problem. Crooza below has solved it perfectly. Yes you have envisioned what I required quite to the point. An alternative to what Crooza has done would be to move the points this time (not by a percentage) but by 1° in each axis. So in a way, we'd be dealing with a vector, where the combination of the movement in each axis would add up by simple pythagoras with the resulting movement in the directon perpendicular to the point.

    Please refer to the below image for elucidation. Note that the red lines are in the direction of the reference axes, and the blue line is in the direction perpendicular to the point. The two red lines are not drawn to scale obviously, for convenience purposes.

    Attachment 642535

    So basically, I would like each point moved 1.4142 degrees ( Sqrt[1^2 + 1^2] ) in the perpendicular direction towards the trend line.

    Would this be approached the same way as what Crooza did?

    Thanks in advance!

  6. #6
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,827

    Re: Moving all data points closer to the trend line on a graph

    I obviously cannot speak for Crooza, but his solution is exclusively using basic arithmetic operators (+,*,^,etc.), so i expect that most of the referencing was to geometry/algebra texts/tutorials (or previous learning in those fields). Once the basic geometry/algebra/arithmetic is recalled, then it is a simple matter to formulate those arithmetic expressions into spreadsheet formulas.

    So in a way, we'd be dealing with a vector,
    In many ways, I would agree that this sort of thing can be more easily thought of in terms of vector algebra. Unfortunately, spreadsheets generally don't have good support for vector algebra*. Often, when I want to use vector algebra for a problem, I end up doing the vector algebra outside of Excel, reduce the final solution to the resulting scalar arithmetic expressions, then enter those arithmetic expressions into Excel.

    So basically, I would like each point moved 1.4142 degrees ( Sqrt[1^2 + 1^2] ) in the perpendicular direction towards the trend line.
    Doesn't this assume that the slope of the perpendiculars is exactly -1 (which maybe assumes that the slope of the regression line is +1)? I note that the slope of the regression line (0.9735) is close to 1. Shouldn't the actual movement take into account the actual slope of the perpendicular and regression lines (movement will not be exactly 1° in each direction)? Or, if the "theoretical" slope of the regression line should be close to 1, should the analysis begin by forcing the slope of the regression line to be exactly 1?

    * -- If you can express your vector algebra problem as complex numbers, spreadsheets do have a battery of functions to work with complex numbers (most of them begin with IM...(), such as the IMSUM() function). A vector problem where the vectors are represented in the complex plane can sometimes be nicely worked out in a spreadsheet using those functions. For example:
    1) Each point can be represented as a complex number roll+yaw*i [In Excel, =COMPLEX(roll,yaw)]
    2) The "movement" vector 1 degree in each direction can be expressed as the complex number 1-i [in Excel, COMPLEX(1,-1)] (You could also make this step more complex so that it takes into account the slope of the regression line to get the desired movement vector)
    3) The "moved" point then becomes the simple sum of those complex numbers roll2+yaw2*i=roll+yaw*i+1-i [in Excel =IMSUM(COMPLEX(roll,yaw),COMPLEX(1,-1))]
    4) In order to chart the points, you will need to separate the real and imaginary parts of the complex number using the REAL() and IMAGINARY() functions.

    At this point, the problem feels more like a math/geometry/algebra problem. Once we understand exactly how you want to do the math/algebra, maybe reduced to arithmetic expressions, then we can work on programming those arithmetic expressions into Excel.

  7. #7
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,827

    Re: Moving all data points closer to the trend line on a graph

    Forum doesn't want to let me edit my post today -- note that I erred. The function for the real portion of a complex number is IMREAL().

  8. #8
    Forum Expert Crooza's Avatar
    Join Date
    10-19-2013
    Location
    Hunter Valley, Australia
    MS-Off Ver
    Excel 2003 /7/10
    Posts
    2,082

    Re: Moving all data points closer to the trend line on a graph

    I simply used geometry. The Linest function gives you the slope (or gradient) and the y-intercept of the line of best fit. Although the chart already had this I set this up in cells D2 and E2 so it is now dynamic and will change if the data set changes.

    The closest distance from a point to a line is the perpendicular distance. The gradient of a perpendicular line will be the negative inverse of the line it is perpendicular to! We know the gradient of the line of best fit is 0.97 so the gradient of the perpendicular line to that is -1/0.97.

    If you know a point and the gradient of a line through that point you can create a formula for the line in the form of y = gradient *x + the y-intercept.

    For each point I calculate the y intercept for the perpendicular line through that point (column F). Once I have two formulas for the two lines I can find where those to lines intersect and that will be the point on the line of best fit which is closest to the data point being investigated.

    In essence then I have two formulas with two unknowns (the x and y coordinate).

    With two formulas and two unknown I can solve for each of these. ( I can show you the arithmetic if you're really interested) but for now I've jumped to the answer

    the x coordinate is equal to the gradient times (the y intercept of the perpendicular line - y intercept of the line of best fit) divided by the (gradient squared + 1)
    or
    x = m(b-c)/(m^2+1)
    once I know the x I can substitute into the line of best fit formula to get y

    From there all I did was proportionally move the point the appropriate percentage point closer to the line (ie between the data point and the intersect point) depending on the percentage entered into D4.

  9. #9
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    Wow! That was amazing! Yes I would definintely like to know the arithmetic and the proof behind the x = m(b-c)/(m^2+1) formula. I love seeing things from first principles. Was great that you made the line dynamic as I wanted to know the new R^2 value. Also, I'd also like to see the grap with the points moved 1° in each axis. In my reply to Mr Shorty above I have explained this and also provided an image. How would your approach be for this? I am thinking it would be the same up untill the end and instead of moving the point by a percentage you would move it along the perpendicular line by 1.4142 degrees ( Sqrt[1^2 + 1^2] )? If excel can move it along the perpendicular line by our chosen value of 1.4142, then we wouldn't even to resort to vector alegbra, if I am not mistaken! I am not sure of how to do this though.

    Can you kindly, attempt this and get back to me. Many thanks!

  10. #10
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    Hi MrShorty

    No the slope of the perpendicular does not have to be -1 and so also the slope of the second perpendicular line (the trend line) does not necessarily have to be +1. This would just be a special (and most simple) case. Both the lines have their own slopes, whatever it may be, and when multiplied together result in -1. There is no forcing of the regression line, that would bestow an incorrect trendline, as I need to work with what I have and don't want to idealize it.

    Thanks for the advice on the complex-numbers approach. It certainly is valid. Apart from the above misunderstanding about the slope value^, I think you have already have it perfectly conceptualized. What remains now is to just reduce it to arithmetic expressions and program it.

    So to sum up, if you took the gradient of the existing trendline (.9375) and gradient of the perpendicular (-1/.9735) you are ready to start the process as you described. I am greatly looking forward to seeing the result of this complex-number operation.

    Many thanks

  11. #11
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,827

    Re: Moving all data points closer to the trend line on a graph

    How much do you need help with?

    The basic geometry is something that is typically taught in secondary school. Tutorials abound around the internet:
    https://www.basic-mathematics.com/di...nd-a-line.html
    https://www.basic-mathematics.com/di...nd-a-line.html
    And Wikipedia, which probably has more detail than is needed: https://en.wikipedia.org/wiki/Distan...oint_to_a_line

    Since I am focusing on approaching this with complex numbers, I figure there are still some details to figure out, but the overall procedure that I have outlined should be good. Is there a particular part of what I outlined that you are having trouble understanding or implementing?

  12. #12
    Forum Expert Crooza's Avatar
    Join Date
    10-19-2013
    Location
    Hunter Valley, Australia
    MS-Off Ver
    Excel 2003 /7/10
    Posts
    2,082

    Re: Moving all data points closer to the trend line on a graph

    I've attached a photo of my working. Hope it helps. I need to read through Mr Shorty's explanation above again. I understand real and complex numbers but not sure I understand the 1 degree thing yet. I glossed over it on my first reading.
    Attached Images Attached Images

  13. #13
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    I would need to go over a few things; from the arithmeticsl to brushing up my complex numbers; to becoming familiar with those excel functions you mentioned. I am in quick need of results just at the moment as I have a project due. So what I will do is flow with moving the data points by the percentage as has been done by Crooza and will return to doing this with complex numbers perhaps a week or two later. Will let you know if I run into any trouble implementing anything. Thanks much for your help!

    Once I get going I should be fine with the entire procedure with only specifics to ask about!

  14. #14
    Registered User
    Join Date
    09-22-2019
    Location
    Auckland, NZ
    MS-Off Ver
    Office 365
    Posts
    7

    Re: Moving all data points closer to the trend line on a graph

    Thanks much for sharing your notes Crooza. Will definitely look into it. How can I change the gradient value of .97351 that you have put in D2. I actually realized that it is better for me to use the gradient of the trendline when it is set (goes through 0,0). When set, the gradient is actually a little less, its .9227. So I want to base your graph of this value and not the .97351.

    I am not being able to change the cell D2. It says "you can't change part of an array". Please let me know how I would be able to change it.

    Cheers!

  15. #15
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,827

    Re: Moving all data points closer to the trend line on a graph

    As explained in the help file for the LINEST() function (https://support.office.com/en-us/art...a-fa7abf772b6d ), you can force the regression through the origin (force b to 0) by putting FALSE/0 in for the optional third argument =LINEST($A$3:$A$34,$B$3:$B$34,0,0)

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 3
    Last Post: 05-18-2019, 09:58 AM
  2. Trend line & data points over wrong bar charts
    By vemix in forum Excel Charting & Pivots
    Replies: 5
    Last Post: 03-17-2016, 12:26 PM
  3. Replies: 1
    Last Post: 08-27-2015, 01:54 AM
  4. forecasting data points through polynomial trend line
    By cjmonks in forum Excel General
    Replies: 1
    Last Post: 06-27-2011, 05:42 PM
  5. Time Graph with trend line is required for three rows of data
    By suryaprasad in forum Excel Charting & Pivots
    Replies: 1
    Last Post: 04-30-2009, 09:01 AM
  6. help adding a trend line to a not-so linear graph, and then moving it
    By jmarcian in forum Access Tables & Databases
    Replies: 4
    Last Post: 03-05-2009, 12:16 AM
  7. how to add data points to a scatter plot with trend line
    By Samantha in forum Excel Charting & Pivots
    Replies: 2
    Last Post: 02-28-2005, 09:06 AM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1