google cloud dataflow - Error during the pipeline execution: exceeds allowed maximum skew -


i executed pipeline friday , has been executing during weekend sunday there lots of below error:

14 jun. 2015 14:40:51 (6f550257718f53da): exception: java.lang.illegalargumentexception: timestamp 2015-06-14t12:40:48.731z exceeds allowed maximum skew. com.google.api.client.repackaged.com.google.common.base.preconditions.checkargument(preconditions.java:119) com.google.api.client.util.preconditions.checkargument(preconditions.java:69) com.google.cloud.dataflow.sdk.util.dofnrunner$dofnprocesscontext.checktimestamp(dofnrunner.java:502) com.google.cloud.dataflow.sdk.util.dofnrunner$dofnprocesscontext.outputwithtimestamp(dofnrunner.java:465) com.xtg.hub.dataflow.stats.common.util$extracttimestampfn.processelement(util.java:62)

in pipeline there 5 minutes fixedwindow , before doing .apply fixedwindow i'm assigning timestamp each element of pcollection:

c.outputwithtimestamp(c.element(), instant.now());

am doing wrong?

thanks in advance.

it's not current supported in dataflow call outputwithtimestamp timestamp less timestamp of input element. it's possible that, due clock skew between workers, setting timestamp instant.now() trying move timestamp backwards.

edit: example, do:

instant = instant.now(); c.outputwithtimestamp(c.element(),                       now.isafter(c.timestamp()) ? : c.timestamp());

Comments

Post a Comment

Popular posts from this blog

javascript - Google App Script ContentService downloadAsFile not working -

i have web app developed using google app script htmlservice , html form, populating excel sheet in google drive using spreadsheetapp . , 1 section calling contentservice download data excel file. function doget(e) { // read excel sheet //getappfile(); // render application html template return htmlservice.createtemplatefromfile('index').evaluate() .settitle('go smart') .setsandboxmode(htmlservice.sandboxmode.iframe); } function downloaddoublequatecsvfile() { var sheetid = propertiesservice.getscriptproperties().getproperty('sheetid'); var ss = spreadsheetapp.openbyid(sheetid).getactivesheet(); //var ss = spreadsheetapp.getactivespreadsheet().getactivesheet(); var maxcolumn = ss.getlastcolumn(); var maxrow = ss.getlastrow(); var data = ss.getrange(1, 1, maxrow, maxcolumn).getvalues(); if (data.length > 1) { var csv = ""; (var row = 0; row < data.length; row++) { (var c...
Read more

javascript - Function overwritting -

is there way overwrite function or event handling in function? using addeventlistener, can have many event handlers on element want var test=document.getelementbyid("div"); test.addeventlistener("click",function(){alert("im first");},false); test.addeventlistener("click",function(){alert("im second");},false); // alerts "im first" , after "im second" but if want overwrite function/event example based on client width example this function one() { alert("first"); } function two() { alert("second"); } window.onresize = function() { var test = document.getelementbyid("div"); if (window.innerwidth > 500) { test.onclick = 1 } else { test.onclick = two; } } is possible in javascript? in case use effective little known approach. addeventlistener can accept object property handleevent event handler. in case it's easy overwr...
Read more

php - Find a regex to take part of Email -

i have following emails need extract part from: bestellnummer xxx: 1 von xxx, 1er pack ------------- anfang der nachricht ------------- foo bar baz foo bar baz // <<<<< need text here ------------- ende der nachricht ------------- ------------- anfang der nachricht ------------- foo bar baz foo bar baz ------------- ende der nachricht ------------- there 0 unlimited occurences of ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- and i'm able extract first part regex: $re = "/------------- .*? -------------.?(.*?).?------------- .*? -------------/s"; but, i'm quite new on learning regex, i'm pretty sure there must better regex extract part (foo bar baz foo bar baz) of text between ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- as can in different languages, i'm using .? to match between hyphens. i...
Read more