r - Seeking on a gz connection is unpredictable -


i'm having trouble seeking around gzfiles in r. here's example:

set.seed(123) m=data.frame(z=runif(10000),x=rnorm(10000)) write.csv(m,"m.csv") system("gzip m.csv") file.info("m.csv.gz")$size [1] 195975

that creates m.csv.gz, r says can seek on, , seek seems agree:

gzf=gzfile("m.csv.gz") open(gzf,"rb") isseekable(gzf) [1] true

now small jumps, , forth, seem work, if try big jump, error:

seek(gzf,10) [1] 10 seek(gzf,20) [1] 10 seek(gzf,10) [1] 20 seek(gzf,1000) [1] 100 warning message: in seek.connection(gzf, 1000) :   seek on gzfile connection returned internal error

however if reset connection , start again, can 1000 if in 100-byte steps:

for(i in seq(100,1000,by=100)){seek(gzf,i)} seek(gzf,na) [1] 1000

r has harsh words on using seek in windows: "use of ‘seek’ on windows discouraged." on linux box (r 3.1.1, 32 bit). similar code in python using gz library works fine, seeking over.

r 3.2.0 more informative:

warning messages: 1: in seek.connection(gzf, 1000) : invalid or incomplete compressed data 2: in seek.connection(gzf, 1000) :   seek on gzfile connection returned internal error

ideas? i've submitted bug report now.

this educated guess: small jumps handled within decoded buffer, when seek more buffer size perform raw seek tries decode gzip in middle of chunk leading decoding error, bug within r library. suggest use skip instead of seek, since underlying library cannot more , won't have impact in performance.

i checked rfc1952 , rfc1951, in gzip can know complete size of file before extracting reading 'members' header , sum isize fields, cannot know how big deflated block without decoding it(the size of each symbol in dictionary), cannot seek common gzip stream.

if want seek gzip must index beforehand.

dictzip library adds headers allow seeking.


Comments

Post a Comment

Popular posts from this blog

c# - Validate object ID from GET to POST -

i have method allows user access comment have made on site. have post method lets them update comment. my normal solution send entire comment model through, let them update it, update on database when post back. involve sending commentid through in hiddenfor , can manipulated. how can verify commentid sent in get method same i'm getting in post , not able alter comment wish? how can verify commentid sent in method same i'm getting in post? basically have validate following things in post - user logged-in user. authenticated users post comments. commentid in post , should valid commentid , should present in database. userid associated logged-in user should same userid associated comment . comment should contain userid column, can check @ time of update. to make sure update happens comment has been sent in get - hold commentid in session , in post action compare commentid value in session .
Read more

node.js - Custom Model Validator SailsJS -

when using custom model validator checks record within model. return within spread doesn't seem end control flow. //post.js model /** * custom validator **/ types: { isuservalid: function(user_id) { var promise = require('bluebird'); promise.all([ user.findone({id: user_id}) ]) .spread(function(user) { console.log(user); if (user === null || user === undefined) { console.log('failed'); return false; }else{ console.log('passed'); return true; } }); } }, my response standard validation failed response. { "error": "e_validation", "status": 400, "summary": "1 attribute invalid", "model": "post", "invalidattributes": { "owner": [ { "rule": "isuservalid", ...
Read more

php - Find a regex to take part of Email -

i have following emails need extract part from: bestellnummer xxx: 1 von xxx, 1er pack ------------- anfang der nachricht ------------- foo bar baz foo bar baz // <<<<< need text here ------------- ende der nachricht ------------- ------------- anfang der nachricht ------------- foo bar baz foo bar baz ------------- ende der nachricht ------------- there 0 unlimited occurences of ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- and i'm able extract first part regex: $re = "/------------- .*? -------------.?(.*?).?------------- .*? -------------/s"; but, i'm quite new on learning regex, i'm pretty sure there must better regex extract part (foo bar baz foo bar baz) of text between ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- as can in different languages, i'm using .? to match between hyphens. i...
Read more