Series of blog posts on data analysis

Ian Ross has posted the first 5 of a possibly indefinite series of literate blog posts on climate data analysis based on:

Daan Crommelin, Observed nondiffusive dynamics in large-scale atmospheric flow (2004)

A particularly interesting citation is:

Egger, J., 2001: Master equation for climatic parameter sets,Climate Dyn., 18, 169–177.

Comments

  • 1.

    Why is it interesting? Okay, it contains the term "master equation" in the title - anything more?

    Comment Source:Why is it interesting? Okay, it contains the term "master equation" in the title - anything more?
  • 2.

    Hello John

    I reviewed the paper by Egger yesterday and the title is rather larger than the contents.

    I like the field-stream visualization he has used, and the idea of CLOUDS of STATES:

    The Master equation provides an excellent visualization of the motion in the phase plane and allows one to predict and analyze the evolution of clouds of states.

    I have no idea how or what data was used to configure the differential equations, but if someone help me with that, the equations are manageable for Mathematica Solvers.

    If I understood him correctly, formation of clouds of points in phase space and study of field-streams along them might shed lights into the wind mechanisms. So use the collected data, form differential equations, but analyze in phase space not in our physical 3D space.

    Again to go from the paper to actual code is a tall order, perhaps better approach is use his concepts but start from scratch

    Dara

    Comment Source:Hello John I reviewed the paper by Egger yesterday and the title is rather larger than the contents. I like the field-stream visualization he has used, and the idea of CLOUDS of STATES: >The Master equation provides an excellent visualization of the motion in the phase plane and allows one to predict and analyze the evolution of clouds of states. I have no idea how or what data was used to configure the differential equations, but if someone help me with that, the equations are manageable for Mathematica Solvers. If I understood him correctly, formation of clouds of points in phase space and study of field-streams along them might shed lights into the wind mechanisms. So use the collected data, form differential equations, but analyze in phase space not in our physical 3D space. Again to go from the paper to actual code is a tall order, perhaps better approach is use his concepts but start from scratch Dara
  • 3.
    edited September 2014

    John wrote:

    it contains the term “master equation” in the title - anything more?

    Crommelin says his method is similar to Egger's but I haven't been back to his paper for a second close reading to see if and how. Egger gives the first example I've come across with the equations and an outline of a rubric for a method of using a master equation in any practical application, not just in climate. He also claims this is its first use in climate modelling. But as all but the first 2 pages are paywalled I have no idea about the rest of the paper.

    Dara wrote:

    to go from the paper to actual code is a tall order, perhaps better approach is use his concepts but start from scratch.

    I was hoping somebody might have access and be able to check out the full paper. I'm excited at the prospect of learning how Ian extracts a working code library from the Crommelin paper. He's also said he would like to try some alternative method for this problem.

    Comment Source:John wrote: > it contains the term “master equation” in the title - anything more? Crommelin says his method is similar to Egger's but I haven't been back to his paper for a second close reading to see if and how. Egger gives the first example I've come across with the equations and an outline of a rubric for a method of using a master equation in any practical application, not just in climate. He also claims this is its first use in climate modelling. But as all but the first 2 pages are paywalled I have no idea about the rest of the paper. Dara wrote: > to go from the paper to actual code is a tall order, perhaps better approach is use his concepts but start from scratch. I was hoping somebody might have access and be able to check out the full paper. I'm excited at the prospect of learning how Ian extracts a working code library from the Crommelin paper. He's also said he would like to try some alternative method for this problem.
  • 4.

    Jim I have access to the paper by J Egger and see nothing about code in it.

    If you write me dara@lossof gen I send you the paper, since it is paid and cannot distribute publicly

    DAra

    Comment Source:Jim I have access to the paper by J Egger and see nothing about code in it. If you write me dara@lossof gen I send you the paper, since it is paid and cannot distribute publicly DAra
  • 5.
    edited September 2014

    Dara, I think you missed that it's Ian (Ross) who's producing code. He appears to have added a write ability to his hnetcdf library. Thanks very much for the offer - saves me a six mile journey. In the Crommelin paper I thought being able to deduce a wind direction from a phase model, if that's what the paper actually said, sounded intriguiging, although despite taking notes, I didn't understand how to do it.

    Comment Source:Dara, I think you missed that it's Ian (Ross) who's producing code. He appears to have added a write ability to his [hnetcdf library](https://github.com/ian-ross/hnetcdf.git). Thanks very much for the offer - saves me a six mile journey. In the Crommelin paper I thought being able to deduce a wind direction from a phase model, if that's what the paper actually said, sounded intriguiging, although despite taking notes, I didn't understand how to do it.
  • 6.
    Hello Jim

    sent you the paper, enjoy.

    EQ 14 and 15 talk about mountain torques and friction torques of Earth's topography against air! I would not believe this unless John puts his stamp of approval. Since the atmosphere has been coupled with sphere of earth for 100s of millions of years if not billions, and if such torques existed they had to be violent! But I am just a lowly programmer, and differ such discussion to experts.

    However if someone like John gave me equations for 16, then I would build the volumetric CLOUD and plot the field-streams like in Fig 2.

    D
    Comment Source:Hello Jim sent you the paper, enjoy. EQ 14 and 15 talk about mountain torques and friction torques of Earth's topography against air! I would not believe this unless John puts his stamp of approval. Since the atmosphere has been coupled with sphere of earth for 100s of millions of years if not billions, and if such torques existed they had to be violent! But I am just a lowly programmer, and differ such discussion to experts. However if someone like John gave me equations for 16, then I would build the volumetric CLOUD and plot the field-streams like in Fig 2. D
  • 7.
    edited September 2014

    Eq. 13 is justified by a claim (which should be fairly easily checkable by somebody who knows the field) that all numerical flow algorithms assume that there is a balance of torques (whatever that looks like visually?) between those due to centrifugal accelerations, equatorial bulge and the gravitational field.

    But your doubts seem to be supported:

    All in all, the analysis of the Master equation shows that the impact of friction and mountain torques can be detected but is small. This result is compatible with Eq. (23) and the discussion. According to Eq. (23) it is the wind terms in Eqs. (13) which are of key importance. We ®nd now that friction and mountain torques should represented in an improved version of Eq. (23).

    There are, however, at best weak indications of such a pattern in Fig. 5. The torques are forcing M1 indeed with \beta_1 =ˆ 0:01 (Table 1). However, the complex eigenvalue \lambda_i ˆ = 0:01 corresponds with a period of 92 days. This eff€ect is almost negligible.

    The paper concludes that planetary Rossby waves contribute most to these torques, in line with Lejanas and Madden (2000).

    Anyway a negative result might still have useful methodology hints..

    Comment Source:Eq. 13 is justified by a claim (which should be fairly easily checkable by somebody who knows the field) that all numerical flow algorithms assume that there is a balance of torques (whatever that looks like visually?) between those due to centrifugal accelerations, equatorial bulge and the gravitational field. But your doubts seem to be supported: > All in all, the analysis of the Master equation shows that the impact of friction and mountain torques can be detected but is small. This result is compatible with Eq. (23) and the discussion. According to Eq. (23) it is the wind terms in Eqs. (13) which are of key importance. We ®nd now that friction and mountain torques should represented in an improved version of Eq. (23). > There are, however, at best weak indications of such a pattern in Fig. 5. The torques are forcing M1 indeed with \beta_1 =ˆ 0:01 (Table 1). However, the complex eigenvalue \lambda_i ˆ = 0:01 corresponds with a period of 92 days. This eff€ect is almost negligible. The paper concludes that planetary Rossby waves contribute most to these torques, in line with Lejanas and Madden (2000). Anyway a negative result might still have useful methodology hints..
  • 8.

    But your doubts seem to be supported

    Hello Jim I would not call that comment DOUBT, I just find these papers making claims and presenting them as almost truth, but in reality there is no proof of any kind.

    This sort of writing I have seen in genome papers where they explain e.g. evolution of flight, when I dig into the references none of the authors' theories holds water on how the flight was evolved i.e. animals running from their predators flapping their limbs of course after 400 million years of that they flew!

    As in the case of genome and evolution writings pushed the wrong un-scientific direction specially in USA via monetization, I find the climate related papers 100% OFF! The conclusion do not match the findings, there is no way I could duplicate their computations and usage of data is FUDGED and thus bias. I don't find one paper I could duplicate neither their math nor their algorithms and usage of data.

    Dara

    So that is why I asked John this week to write us a guideline on scientific investigation and how we should conduct ourselves when doing scientific endeavours

    D

    Comment Source:>But your doubts seem to be supported Hello Jim I would not call that comment DOUBT, I just find these papers making claims and presenting them as almost truth, but in reality there is no proof of any kind. This sort of writing I have seen in genome papers where they explain e.g. evolution of flight, when I dig into the references none of the authors' theories holds water on how the flight was evolved i.e. animals running from their predators flapping their limbs of course after 400 million years of that they flew! As in the case of genome and evolution writings pushed the wrong un-scientific direction specially in USA via monetization, I find the climate related papers 100% OFF! The conclusion do not match the findings, there is no way I could duplicate their computations and usage of data is FUDGED and thus bias. I don't find one paper I could duplicate neither their math nor their algorithms and usage of data. Dara So that is why I asked John this week to write us a guideline on scientific investigation and how we should conduct ourselves when doing scientific endeavours D
  • 9.

    Again to be clear, I am not saying I am right, I am saying I do not know what to do to investigate properly. I am ignorant of how a correct scientific conduct should be dealing with these climate data

    Comment Source:Again to be clear, I am not saying I am right, I am saying I do not know what to do to investigate properly. I am ignorant of how a correct scientific conduct should be dealing with these climate data
  • 10.

    Back to Egger's paper I liked his computation idea of CLOUDS OF STATES and their animated field-streams in phase space I assume, now that is something we could use to investigate.

    Comment Source:Back to Egger's paper I liked his computation idea of CLOUDS OF STATES and their animated field-streams in phase space I assume, now that is something we could use to investigate.
  • 11.

    Hello Jim

    In Crommelin paper EQ 4.1-4 indicates doubly stochastic matrices of positive or 0 entries. This means that infinite power of the matrix is always convergent or there is always a stable state.

    The rest of the results in the paper follows accordingly.

    But if a process has such matrices, and from my numerical playing around with stochastic matrices, they converge very fast i.e. less than 20 powers. Therefore the process quickly in seconds or minutes or hours or days should converge and into a stable state and no reason to move out of that sate.

    So the weather systems should be highly stable.

    Either the model using stochastic matrices is not applicable in full or there is another process in nature that RESETS these matrices so the stability state is avoided or less frequent. Here is where I need to learn more from John

    Dara

    Comment Source:Hello Jim In Crommelin paper EQ 4.1-4 indicates doubly stochastic matrices of positive or 0 entries. This means that infinite power of the matrix is always convergent or there is always a stable state. The rest of the results in the paper follows accordingly. But if a process has such matrices, and from my numerical playing around with stochastic matrices, they converge very fast i.e. less than 20 powers. Therefore the process quickly in seconds or minutes or hours or days should converge and into a stable state and no reason to move out of that sate. So the weather systems should be highly stable. Either the model using stochastic matrices is not applicable in full or there is another process in nature that RESETS these matrices so the stability state is avoided or less frequent. Here is where I need to learn more from John Dara
  • 12.

    More on Crommelin paper:

    John wrote a beautiful historical expose of physics theories and proposed that their common mechanism is Exp map from the Lie Algebra space to Lie group (I might be misquoting).

    So the stochastic matrices' Lie Algebra is known and it is I believe of 2nd order i.e. if the dimension of matrix is n then Lie Algebra is n^2 - n ie huge.

    The BCH formula governing the Exp map indicates a complicated difficult to compute formula and indeed shows how complicated such mappings and the underlying systems are

    Comment Source:More on Crommelin paper: John wrote a beautiful historical expose of physics theories and proposed that their common mechanism is Exp map from the Lie Algebra space to Lie group (I might be misquoting). So the stochastic matrices' Lie Algebra is known and it is I believe of 2nd order i.e. if the dimension of matrix is n then Lie Algebra is n^2 - n ie huge. The BCH formula governing the Exp map indicates a complicated difficult to compute formula and indeed shows how complicated such mappings and the underlying systems are
  • 13.

    Dara wrote:

    So that is why I asked John this week to write us a guideline on scientific investigation and how we should conduct ourselves when doing scientific endeavours.

    Wow, writing that could be a huge job! I can imagine a book on how to do science, but I don't have time to write that book. I think it would be easier for me to say what we should try to do with this El Niño project. Even that will be quite difficult for me.

    Comment Source:Dara wrote: > So that is why I asked John this week to write us a guideline on scientific investigation and how we should conduct ourselves when doing scientific endeavours. Wow, writing that could be a huge job! I can imagine a book on how to do science, but I don't have time to write that book. I think it would be easier for me to say what we should try to do with this El Niño project. Even that will be quite difficult for me.
Sign In or Register to comment.