Is there a limit on the number of side outputs in Google Cloud Dataflow? -


we have cloud dataflow job takes in bigquery table, transforms , writes each record out different table depending on month/year in timestamp record. when run our job on table 12 months of data there should 12 output tables. first month main output , other 11 months side outputs.

we have found job fail when run on 10 or more months(9 side outputs).

is limit on cloud dataflow or bug?

i noticed in execution graph when running more 8 side outputs of outputs said "running" didn't seem writing records.

here of our job ids:

2015-06-14_23_58_06-14457541029573485807 (8 side outputs - passed)

2015-06-14_23_48_43-15277609445992188388 (9 side outputs - failed)

2015-06-14_23_11_46-10500077558949649888 (7 side outputs - passed)

2015-06-14_22_38_48-1428211312699949403 (3 side outputs - passed)

2015-06-14_21_44_27-16273252623089185131 (11 side outputs - failed)

this code processes data. there no caching involved. (tressoutputmanager holds cache of tupletag<tablerow>)

public class tressdenormalizationdofn extends dofn<tablerow, tablerow> {     @inject     @named("tress.mappers")     private set<cptmapper> mappers;     @inject     private tressoutputmanager tuples;      @override     public void processelement(processcontext c) throws exception {         tablerow row = c.element().clone();         (cptmapper mapper : mappers) {             string mapped = mapper.map((string) row.get("event"));             if (mapped != null) {                 row.set(mapper.getid(), mapped);             }         }         // places record in correct month based on time stamp         string timestamp = (string) row.get("time_local");         if(timestamp != null){             timestamp = timestamp.substring(0, 7).replaceall("-", "_");              if (tuples.ismainoutput(timestamp)) {                 c.output(row);             } else {                 c.sideoutput(tuples.gettuple(timestamp), row);             }         }     } }


Comments

Post a Comment

Popular posts from this blog

javascript - Google App Script ContentService downloadAsFile not working -

i have web app developed using google app script htmlservice , html form, populating excel sheet in google drive using spreadsheetapp . , 1 section calling contentservice download data excel file. function doget(e) { // read excel sheet //getappfile(); // render application html template return htmlservice.createtemplatefromfile('index').evaluate() .settitle('go smart') .setsandboxmode(htmlservice.sandboxmode.iframe); } function downloaddoublequatecsvfile() { var sheetid = propertiesservice.getscriptproperties().getproperty('sheetid'); var ss = spreadsheetapp.openbyid(sheetid).getactivesheet(); //var ss = spreadsheetapp.getactivespreadsheet().getactivesheet(); var maxcolumn = ss.getlastcolumn(); var maxrow = ss.getlastrow(); var data = ss.getrange(1, 1, maxrow, maxcolumn).getvalues(); if (data.length > 1) { var csv = ""; (var row = 0; row < data.length; row++) { (var c...
Read more

javascript - Function overwritting -

is there way overwrite function or event handling in function? using addeventlistener, can have many event handlers on element want var test=document.getelementbyid("div"); test.addeventlistener("click",function(){alert("im first");},false); test.addeventlistener("click",function(){alert("im second");},false); // alerts "im first" , after "im second" but if want overwrite function/event example based on client width example this function one() { alert("first"); } function two() { alert("second"); } window.onresize = function() { var test = document.getelementbyid("div"); if (window.innerwidth > 500) { test.onclick = 1 } else { test.onclick = two; } } is possible in javascript? in case use effective little known approach. addeventlistener can accept object property handleevent event handler. in case it's easy overwr...
Read more

php - Find a regex to take part of Email -

i have following emails need extract part from: bestellnummer xxx: 1 von xxx, 1er pack ------------- anfang der nachricht ------------- foo bar baz foo bar baz // <<<<< need text here ------------- ende der nachricht ------------- ------------- anfang der nachricht ------------- foo bar baz foo bar baz ------------- ende der nachricht ------------- there 0 unlimited occurences of ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- and i'm able extract first part regex: $re = "/------------- .*? -------------.?(.*?).?------------- .*? -------------/s"; but, i'm quite new on learning regex, i'm pretty sure there must better regex extract part (foo bar baz foo bar baz) of text between ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- as can in different languages, i'm using .? to match between hyphens. i...
Read more