Optimising a [large] look-up table within a loop, in R












0















I would really appreciate any help in optimising the code below. For the number of iterations I need to run, it currently takes way to long.



Basically, I'm looking for the optimal set of model parameters for estimating a variable of interest. I'm simulating a set of model parameters and a variable of interest, using each of them (in a loop) to model the signal of interest, then compare each of the modeled signal to the observed signal, and where they are minimised select one of the (simulated) values we're trying to predict. Then, a RMSE function is used to select which set of model parameters offers the smallest error compared to the observed values.



Many Thanks



Code



#Empty variable

 est_hold <- NULL



# Model Parameters (simulated and/or estimated via NLS earlier)

    beta <- seq(0.001,0.2,0.01)

    sig.g <- seq(0.028, 0.038,length.out = 20)

    sig.v <- seq(0.6, 0.8,length.out = 20)

    params <- data.frame(rbind(beta,sig.g,sig.v))



# Simulated variable of interest (Dependant variable)

sim_var <- seq(1,100,0.1)



# Observed signal (Independant variable)

obs_sig <- seq(0.0001,0.8,length.out = 100000)



# Observed variable of interest (Dependant variable)

obs_var2 <- seq(1,100,length.out = 100000)





min.func <- function(x) {sim_var[which.min(x)]}



system.time({

  for(c in 1:ncol(params)){



    # Modelled signal (Independant) 

    model_var <- params[2,c]*exp(-params[1,c]*sim_var)+params[3,c]*(1-exp(-params[1,c]*sim_var))



    # Calculate difference between each modelled value (using simulated params) and each observed value

    diff <- t(apply(as.data.frame(model_var),1,"-",obs_sig)) 

    diff <- abs(diff)



    # Apply function, to select a simulated dependant value where differences are minimised

    est <- apply(diff,2,min.func)

    rm(diff)



    est_hold <- cbind(est_hold,unlist(est))



  }})



# RMSE function, to calculate RMSE between observed dependant variable and the modelled dependant variable (using simulated params)

rmse.fun = function(x){sqrt(sum((obs_var2-x)^2)/length(obs_var2))}



# Apply the RMSE function to the matrix of modelled values 

# (where each column represents modelled values using each of the simulated params)

sim_rmse <- apply(est_hold,2,rmse.fun)



# Find the position of the lowest RMSE

ind <- which.min(sim_rmse)



#Use this position to subset the parameters (...and continue with script/modelling)

final_params <- params[ind]





r optimization simulation lookup







share|improve this question























  • the data.table package could be interesting for you,..

    – BigDataScientist
    Nov 21 '18 at 22:34











  • I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

    – user2738526
    Nov 22 '18 at 6:00
















0















I would really appreciate any help in optimising the code below. For the number of iterations I need to run, it currently takes way to long.



Basically, I'm looking for the optimal set of model parameters for estimating a variable of interest. I'm simulating a set of model parameters and a variable of interest, using each of them (in a loop) to model the signal of interest, then compare each of the modeled signal to the observed signal, and where they are minimised select one of the (simulated) values we're trying to predict. Then, a RMSE function is used to select which set of model parameters offers the smallest error compared to the observed values.



Many Thanks



Code



#Empty variable

 est_hold <- NULL



# Model Parameters (simulated and/or estimated via NLS earlier)

    beta <- seq(0.001,0.2,0.01)

    sig.g <- seq(0.028, 0.038,length.out = 20)

    sig.v <- seq(0.6, 0.8,length.out = 20)

    params <- data.frame(rbind(beta,sig.g,sig.v))



# Simulated variable of interest (Dependant variable)

sim_var <- seq(1,100,0.1)



# Observed signal (Independant variable)

obs_sig <- seq(0.0001,0.8,length.out = 100000)



# Observed variable of interest (Dependant variable)

obs_var2 <- seq(1,100,length.out = 100000)





min.func <- function(x) {sim_var[which.min(x)]}



system.time({

  for(c in 1:ncol(params)){



    # Modelled signal (Independant) 

    model_var <- params[2,c]*exp(-params[1,c]*sim_var)+params[3,c]*(1-exp(-params[1,c]*sim_var))



    # Calculate difference between each modelled value (using simulated params) and each observed value

    diff <- t(apply(as.data.frame(model_var),1,"-",obs_sig)) 

    diff <- abs(diff)



    # Apply function, to select a simulated dependant value where differences are minimised

    est <- apply(diff,2,min.func)

    rm(diff)



    est_hold <- cbind(est_hold,unlist(est))



  }})



# RMSE function, to calculate RMSE between observed dependant variable and the modelled dependant variable (using simulated params)

rmse.fun = function(x){sqrt(sum((obs_var2-x)^2)/length(obs_var2))}



# Apply the RMSE function to the matrix of modelled values 

# (where each column represents modelled values using each of the simulated params)

sim_rmse <- apply(est_hold,2,rmse.fun)



# Find the position of the lowest RMSE

ind <- which.min(sim_rmse)



#Use this position to subset the parameters (...and continue with script/modelling)

final_params <- params[ind]





r optimization simulation lookup







share|improve this question























  • the data.table package could be interesting for you,..

    – BigDataScientist
    Nov 21 '18 at 22:34











  • I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

    – user2738526
    Nov 22 '18 at 6:00














0












0








0








I would really appreciate any help in optimising the code below. For the number of iterations I need to run, it currently takes way to long.



Basically, I'm looking for the optimal set of model parameters for estimating a variable of interest. I'm simulating a set of model parameters and a variable of interest, using each of them (in a loop) to model the signal of interest, then compare each of the modeled signal to the observed signal, and where they are minimised select one of the (simulated) values we're trying to predict. Then, a RMSE function is used to select which set of model parameters offers the smallest error compared to the observed values.



Many Thanks



Code



#Empty variable

 est_hold <- NULL



# Model Parameters (simulated and/or estimated via NLS earlier)

    beta <- seq(0.001,0.2,0.01)

    sig.g <- seq(0.028, 0.038,length.out = 20)

    sig.v <- seq(0.6, 0.8,length.out = 20)

    params <- data.frame(rbind(beta,sig.g,sig.v))



# Simulated variable of interest (Dependant variable)

sim_var <- seq(1,100,0.1)



# Observed signal (Independant variable)

obs_sig <- seq(0.0001,0.8,length.out = 100000)



# Observed variable of interest (Dependant variable)

obs_var2 <- seq(1,100,length.out = 100000)





min.func <- function(x) {sim_var[which.min(x)]}



system.time({

  for(c in 1:ncol(params)){



    # Modelled signal (Independant) 

    model_var <- params[2,c]*exp(-params[1,c]*sim_var)+params[3,c]*(1-exp(-params[1,c]*sim_var))



    # Calculate difference between each modelled value (using simulated params) and each observed value

    diff <- t(apply(as.data.frame(model_var),1,"-",obs_sig)) 

    diff <- abs(diff)



    # Apply function, to select a simulated dependant value where differences are minimised

    est <- apply(diff,2,min.func)

    rm(diff)



    est_hold <- cbind(est_hold,unlist(est))



  }})



# RMSE function, to calculate RMSE between observed dependant variable and the modelled dependant variable (using simulated params)

rmse.fun = function(x){sqrt(sum((obs_var2-x)^2)/length(obs_var2))}



# Apply the RMSE function to the matrix of modelled values 

# (where each column represents modelled values using each of the simulated params)

sim_rmse <- apply(est_hold,2,rmse.fun)



# Find the position of the lowest RMSE

ind <- which.min(sim_rmse)



#Use this position to subset the parameters (...and continue with script/modelling)

final_params <- params[ind]





r optimization simulation lookup







share|improve this question














I would really appreciate any help in optimising the code below. For the number of iterations I need to run, it currently takes way to long.



Basically, I'm looking for the optimal set of model parameters for estimating a variable of interest. I'm simulating a set of model parameters and a variable of interest, using each of them (in a loop) to model the signal of interest, then compare each of the modeled signal to the observed signal, and where they are minimised select one of the (simulated) values we're trying to predict. Then, a RMSE function is used to select which set of model parameters offers the smallest error compared to the observed values.



Many Thanks



Code



#Empty variable

 est_hold <- NULL



# Model Parameters (simulated and/or estimated via NLS earlier)

    beta <- seq(0.001,0.2,0.01)

    sig.g <- seq(0.028, 0.038,length.out = 20)

    sig.v <- seq(0.6, 0.8,length.out = 20)

    params <- data.frame(rbind(beta,sig.g,sig.v))



# Simulated variable of interest (Dependant variable)

sim_var <- seq(1,100,0.1)



# Observed signal (Independant variable)

obs_sig <- seq(0.0001,0.8,length.out = 100000)



# Observed variable of interest (Dependant variable)

obs_var2 <- seq(1,100,length.out = 100000)





min.func <- function(x) {sim_var[which.min(x)]}



system.time({

  for(c in 1:ncol(params)){



    # Modelled signal (Independant) 

    model_var <- params[2,c]*exp(-params[1,c]*sim_var)+params[3,c]*(1-exp(-params[1,c]*sim_var))



    # Calculate difference between each modelled value (using simulated params) and each observed value

    diff <- t(apply(as.data.frame(model_var),1,"-",obs_sig)) 

    diff <- abs(diff)



    # Apply function, to select a simulated dependant value where differences are minimised

    est <- apply(diff,2,min.func)

    rm(diff)



    est_hold <- cbind(est_hold,unlist(est))



  }})



# RMSE function, to calculate RMSE between observed dependant variable and the modelled dependant variable (using simulated params)

rmse.fun = function(x){sqrt(sum((obs_var2-x)^2)/length(obs_var2))}



# Apply the RMSE function to the matrix of modelled values 

# (where each column represents modelled values using each of the simulated params)

sim_rmse <- apply(est_hold,2,rmse.fun)



# Find the position of the lowest RMSE

ind <- which.min(sim_rmse)



#Use this position to subset the parameters (...and continue with script/modelling)

final_params <- params[ind]





r optimization simulation lookup




r optimization simulation lookup






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 21 '18 at 21:00









Sentinel1bSentinel1b

155




155













  • the data.table package could be interesting for you,..

    – BigDataScientist
    Nov 21 '18 at 22:34











  • I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

    – user2738526
    Nov 22 '18 at 6:00



















  • the data.table package could be interesting for you,..

    – BigDataScientist
    Nov 21 '18 at 22:34











  • I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

    – user2738526
    Nov 22 '18 at 6:00

















the data.table package could be interesting for you,..

– BigDataScientist
Nov 21 '18 at 22:34





the data.table package could be interesting for you,..

– BigDataScientist
Nov 21 '18 at 22:34













I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

– user2738526
Nov 22 '18 at 6:00





I'd recommend profvis package as a great way to see what the slowest part of the code is, so you can spend your time trying to fix that slowest bits first

– user2738526
Nov 22 '18 at 6:00












0






active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420411%2foptimising-a-large-look-up-table-within-a-loop-in-r%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420411%2foptimising-a-large-look-up-table-within-a-loop-in-r%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

How to change which sound is reproduced for terminal bell?

Image
3 Recently upgrade to 18.10 from 18.04, and using the terminal is becoming annoying: the new alert sound is a drip ( /usr/share/sounds/gnome/default/alerts/drip.ogg ), and the previous one is still located at ( /usr/share/sounds/ubuntu/stereo/bell.ogg ). My issue is that at the gnome-terminal, when autocompleting or pressing left BOTH sounds are reproduced, at the same time. I want to just have ubuntu/stereo/bell.ogg active, not both . At the moment I have been only able to reduce the volume of the alerts (but still both are played), or turn the alerts down (but I dislike both "solutions"). gnome-terminal 18.10 teminal-bell share | improve this question ...
Read more

Can I use Tabulator js library in my java Spring + Thymeleaf project?

Image
0 I want to add Tabulator js to my java project. I added tabulator.min.js, tabulator.min.css to resources folder. After that added to my html page these dependencies as recommending in readme.md. <link rel="stylesheet" th:href="@{/css/tabulator.min.css}"> <script type="text/javascript" th:src="@{/js/tabulator.min.js}"></script> Now my tables view all elements, that I have in the html page, but not correct. In the documentation of Tabulator is example how to create header and table, var table = new Tabulator("#example-table", { columns:[ {title:"Name", field:"name", sorter:"string", width:200, editor:true}, {title:"Age", field:"age", sorter:"...
Read more

Title Spacing in Bjornstrup Chapter, Removing Chapter Number From Contents

Image
up vote 5 down vote favorite 2 I would like to make it so that the vertical space below the title of each chapter is the same when using the Bjornstrup package (i.e. so that the descenders do not affect the spacing - as shown by the red arrow in the picture below). I think I'm fairly happy with the spacing around "A chapter", so I'd really like titles without descenders (e.g. "Contents") to have the same vertical spacing as "A chapter" below the title. I would also like to remove the large "0" in the contents title. Any help appreciated, as always. documentclass[11pt,a4paper]{book} % Packages[![enter image description here][1]][1] usepackage[left=3.5cm, right=3.5cm, top=4.3cm, bottom=4.25cm]{geometry} usepackage[utf8]{inputenc} usepackage{libertine} usepackage[libertine]{newtx...
Read more