Automated webtasks on javascript enabled webpages, using Java -


i'm using java.

basically i'm trying achieve able run automated tasks on web. example login site, stuff there, fill out forms, logout etc.

i've managed (login , send comments) on simple sites no javascript code present (only html, html forms), sending basic post , messages using httpurlconnection class.

the problem came across site, uses javascript code generate session cookies (on client side). happens when access site, javascript code gets executed in browser, generating session cookies. cookies sent each request, without them, site won't function. (it site http://www.jofogas.hu/ )

i cannot cookies, because not know how find out javascript code gets executed , don't know how run javascript code in java , use results of code.

to answer how:

first saw set-cookie header did not set cookie named ses , cookie magically appears. know in javascript, set cookie call document.cookie. grabbed source code, auto indent it, , search document.cookie. fall on setnewsessioncookie , can see create cookie named ses.

function setnewsessioncookie() {     var b = get_vid();     var d;     var e;     var a;     var c = new date();     c.settime(c.gettime());     = getcookie("ses");     if (!a) {         b = get_vid();         if (!b) {             return         }         d = (new date()).gettime();         = b + "_" + d     }     e = new date(c.gettime() + (1000 * 60 * 15));     document.cookie = name + "ses=" + + ";expires=" + e.togmtstring() }

this cookie set value of variable a set get_vid().

function get_vid() {     var b;     var = getcookie("vsid");     if (!a) {         return false     }     if (a.search(",") == "-1") {         b = base64.decode(a)     } else {         b =     }     return b.split(",")[0] }

tl;dr: function vsid cookie (which send set-cookie header), base64 decode it, , return part before coma. next ses cookie created appending "_" + (new date()).gettime()

in case:

getcookie("vsid") // "odk5njewmzesmtqzndmwndm4ng==" base64.decode(getcookie("vsid")) // "89961031,1434304386" getcookie("ses") // "89961031_1434304048406"

i warn legal issue on web scrapping, may against website terms of use (which not in language speak ...).

edit: question:

automated webtasks on javascript enabled webpages, using java

you may interessed in selenium project , java driver


Comments

Post a Comment

Popular posts from this blog

c# - Validate object ID from GET to POST -

i have method allows user access comment have made on site. have post method lets them update comment. my normal solution send entire comment model through, let them update it, update on database when post back. involve sending commentid through in hiddenfor , can manipulated. how can verify commentid sent in get method same i'm getting in post , not able alter comment wish? how can verify commentid sent in method same i'm getting in post? basically have validate following things in post - user logged-in user. authenticated users post comments. commentid in post , should valid commentid , should present in database. userid associated logged-in user should same userid associated comment . comment should contain userid column, can check @ time of update. to make sure update happens comment has been sent in get - hold commentid in session , in post action compare commentid value in session .
Read more

node.js - Custom Model Validator SailsJS -

when using custom model validator checks record within model. return within spread doesn't seem end control flow. //post.js model /** * custom validator **/ types: { isuservalid: function(user_id) { var promise = require('bluebird'); promise.all([ user.findone({id: user_id}) ]) .spread(function(user) { console.log(user); if (user === null || user === undefined) { console.log('failed'); return false; }else{ console.log('passed'); return true; } }); } }, my response standard validation failed response. { "error": "e_validation", "status": 400, "summary": "1 attribute invalid", "model": "post", "invalidattributes": { "owner": [ { "rule": "isuservalid", ...
Read more

php - Find a regex to take part of Email -

i have following emails need extract part from: bestellnummer xxx: 1 von xxx, 1er pack ------------- anfang der nachricht ------------- foo bar baz foo bar baz // <<<<< need text here ------------- ende der nachricht ------------- ------------- anfang der nachricht ------------- foo bar baz foo bar baz ------------- ende der nachricht ------------- there 0 unlimited occurences of ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- and i'm able extract first part regex: $re = "/------------- .*? -------------.?(.*?).?------------- .*? -------------/s"; but, i'm quite new on learning regex, i'm pretty sure there must better regex extract part (foo bar baz foo bar baz) of text between ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- as can in different languages, i'm using .? to match between hyphens. i...
Read more