Rectangle 27 10

You can do it, but it's more involved than I think you realize. Neither curl nor wget have Javascript engines, so you'll need something that has one.

nice, i didnt knew that there where such things :) I will also update my screenshot script which currently goes over some edges then using phantomjs!

html - get sourcecode after javascript execution with curl - Stack Ove...

javascript html curl
Rectangle 27 11

Although there are no specific NodeJS bindings for cURL, we can still issue cURL requests via the command line interface. NodeJS comes with the child_process module which easily allows us to start processes and read their output. Doing so is fairly straight forward. We just need to import the exec method from the child_process module and call it. The first parameter is the command we want to execute and the second is a callback function that accepts error, stdout, stderr.

var util = require('util');
var exec = require('child_process').exec;

var command = 'curl -sL -w "%{http_code} %{time_total}\\n" "http://query7.com" -o /dev/null'

child = exec(command, function(error, stdout, stderr){

console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);

if(error !== null)
{
    console.log('exec error: ' + error);
}

});

Why use curl from the command line when you can use http.request or node.js directly

In some proxy configurations, it's maddening to try to get node to negotiate the proxy correctly when curl works without trouble. This is a decent workaround for those who have specific needs.

See this answer as well, for attaching a listener to the child process if you want to process the returned results from curl. stackoverflow.com/questions/14332721/

javascript - how to do this curl operation in node.js - Stack Overflow

javascript node.js asynchronous
Rectangle 27 11

Although there are no specific NodeJS bindings for cURL, we can still issue cURL requests via the command line interface. NodeJS comes with the child_process module which easily allows us to start processes and read their output. Doing so is fairly straight forward. We just need to import the exec method from the child_process module and call it. The first parameter is the command we want to execute and the second is a callback function that accepts error, stdout, stderr.

var util = require('util');
var exec = require('child_process').exec;

var command = 'curl -sL -w "%{http_code} %{time_total}\\n" "http://query7.com" -o /dev/null'

child = exec(command, function(error, stdout, stderr){

console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);

if(error !== null)
{
    console.log('exec error: ' + error);
}

});

Why use curl from the command line when you can use http.request or node.js directly

In some proxy configurations, it's maddening to try to get node to negotiate the proxy correctly when curl works without trouble. This is a decent workaround for those who have specific needs.

See this answer as well, for attaching a listener to the child process if you want to process the returned results from curl. stackoverflow.com/questions/14332721/

javascript - how to do this curl operation in node.js - Stack Overflow

javascript node.js asynchronous
Rectangle 27 1

The meta tag has the content attribute which can be used to specify a timeout. However, in JavaScript it is nearly impossible to detect when the redirect will happen as there are infinite ways to write it. At least not without executing it. Example:

var t = 100;
setTimeout(function () {
    window.location.href='http://location2.com/';
}, t * 2);

redirect - Meta tag and javascript browser redirection - which has pri...

javascript redirect curl meta-tags
Rectangle 27 49

So what can you do?

This is one problem that a lot of people have encountered. As pst points out, the bot can just submit information directly to the server, bypassing the javascript (see simple utilities like cURL and Postman). Many bots are capable of consuming and interacting with the javascript now. Hari krishnan points out the use of captcha, the most prevalent and successful of which (to my knowledge) is reCaptcha. But captchas have their problems and are discouraged by the World-Wide Web compendium, mostly for reasons of ineffectiveness and inaccessibility.

And lest we forget, an attacker can always deploy human intelligence to defeat a captcha. There are stories of attackers paying for people to crack captchas for spamming purposes without the workers realizing they're participating in illegal activities. Amazon offers a service called Mechanical Turk that tackles things like this. Amazon would strenuously object if you were to use their service for malicious purposes, and it has the downside of costing money and creating a paper trail. However, there are more erhm providers out there who would harbor no such objections.

My favorite mechanism is a hidden checkbox. Make it have a label like 'Do you agree to the terms and conditions of using our services?' perhaps even with a link to some serious looking terms. But you default it to unchecked and hide it through css: position it off page, put it in a container with a zero height or zero width, position a div over top of it with a higher z-index. Roll your own mechanism here and be creative.

The secret is that no human will see the checkbox, but most bots fill forms by inspecting the page and manipulating it directly, not through actual vision. Therefore, any form that comes in with that checkbox value set allows you to know it wasn't filled by a human. This technique is called a bot trap. The rule of thumb for the type of auto-form filling bots is that if a human has to intercede to overcome an individual site, then they've lost all the money (in the form of their time) they would have made by spreading their spam advertisements.

(The previous rule of thumb assumes you're protecting a forum or comment form. If actual money or personal information is on the line, then you need more security than just one heuristic. This is still security through obscurity, it just turns out that obscurity is enough to protect you from casual, scripted attacks. Don't deceive yourself into thinking this secures your website against all attacks.)

The other half of the secret is keeping it. Do not alter the response in any way if the box is checked. Show the same confirmation, thank you, or whatever message or page afterwards. That will prevent the bot from knowing it has been rejected.

I am also a fan of the timing method. You have to implement it entirely on the server side. Track the time the page was served in a persistent way (essentially the session) and compare it against the time the form submission comes in. This prevents forgery or even letting the bot know it's being timed - if you make the served time a part of the form or javascript, then you've let them know you're on to them, inviting a more sophisticated approach.

Again though, just silently discard the request while serving the same thank you page (or introduce a delay in responding to the spam form, if you want to be vindictive - this may not keep them from overwhelming your server and it may even let them overwhelm you faster, by keeping more connections open longer. At that point, you need a hardware solution, a firewall on a load balancer setup).

There are a lot of resources out there about delaying server responses to slow down attackers, frequently in the form of brute-force password attempts. This IT Security question looks like a good starting point.

I had been thinking about updating this question for a while regarding the topic of computer vision and form submission. An article surfaced recently that pointed me to this blog post by Steve Hickson, a computer vision enthusiast. Snapchat (apparently some social media platform? I've never used it, feeling older every day...) launched a new captcha-like system where you have to identify pictures (cartoons, really) which contain a ghost. Steve proved that this doesn't verify squat about the submitter, because in typical fashion, computers are better and faster at identifying this simple type of image.

It's not hard to imagine extending a similar approach to other Captcha types. I did a search and found these links interesting as well:

Oh, and we'd hardly be complete without an obligatory XKCD comic.

Wow, thank you for the information. I have read up on ways to prevent bots and most suggest CAPTCHA's, but lately I've been reading people say CAPTCHA's are not going to be around in the near future. THis gives me information that I can research, thank you.

I wouldn't say they won't be around in the near future. In my opinion, they have enough of a downside that they're falling out of favor for widespread use. There are plenty of stories (or rants) of Captchas making it way harder for legitimate users and not even stopping 100% of bot traffic. For senstivite applications, a level of hard-ness is acceptable, but if it's a small application or especially one where you benefit more than the user from them completing your form (e.g. feedback, or a business model with heavy competition), Captchas can cause you more problems than they solve.

In case of the register form, should I apply this measure too? Patrick M, check my profile, please.

In general, yes, you want to protect every input form with some kind of human detection. Registration forms typically require email verification; not for bot detection but to verify that you have some way to contact the user. If it's registration for an email service, well, check out what gmail does when you create a new account (and they have spam detection built into the sending protocol). If it's registration for a public forum, then absolutely use as much bot detection as you can, because (in my experience) that attracts the most bots looking for easy ways to spam.

I am not a lawyer and this is not legal advice. You may still violate any number of laws for user protections and privacy even if you provide the utmost bot detection.

javascript - Preventing bot form submission - Stack Overflow

javascript jquery forms security spam
Rectangle 27 49

So what can you do?

This is one problem that a lot of people have encountered. As pst points out, the bot can just submit information directly to the server, bypassing the javascript (see simple utilities like cURL and Postman). Many bots are capable of consuming and interacting with the javascript now. Hari krishnan points out the use of captcha, the most prevalent and successful of which (to my knowledge) is reCaptcha. But captchas have their problems and are discouraged by the World-Wide Web compendium, mostly for reasons of ineffectiveness and inaccessibility.

And lest we forget, an attacker can always deploy human intelligence to defeat a captcha. There are stories of attackers paying for people to crack captchas for spamming purposes without the workers realizing they're participating in illegal activities. Amazon offers a service called Mechanical Turk that tackles things like this. Amazon would strenuously object if you were to use their service for malicious purposes, and it has the downside of costing money and creating a paper trail. However, there are more erhm providers out there who would harbor no such objections.

My favorite mechanism is a hidden checkbox. Make it have a label like 'Do you agree to the terms and conditions of using our services?' perhaps even with a link to some serious looking terms. But you default it to unchecked and hide it through css: position it off page, put it in a container with a zero height or zero width, position a div over top of it with a higher z-index. Roll your own mechanism here and be creative.

The secret is that no human will see the checkbox, but most bots fill forms by inspecting the page and manipulating it directly, not through actual vision. Therefore, any form that comes in with that checkbox value set allows you to know it wasn't filled by a human. This technique is called a bot trap. The rule of thumb for the type of auto-form filling bots is that if a human has to intercede to overcome an individual site, then they've lost all the money (in the form of their time) they would have made by spreading their spam advertisements.

(The previous rule of thumb assumes you're protecting a forum or comment form. If actual money or personal information is on the line, then you need more security than just one heuristic. This is still security through obscurity, it just turns out that obscurity is enough to protect you from casual, scripted attacks. Don't deceive yourself into thinking this secures your website against all attacks.)

The other half of the secret is keeping it. Do not alter the response in any way if the box is checked. Show the same confirmation, thank you, or whatever message or page afterwards. That will prevent the bot from knowing it has been rejected.

I am also a fan of the timing method. You have to implement it entirely on the server side. Track the time the page was served in a persistent way (essentially the session) and compare it against the time the form submission comes in. This prevents forgery or even letting the bot know it's being timed - if you make the served time a part of the form or javascript, then you've let them know you're on to them, inviting a more sophisticated approach.

Again though, just silently discard the request while serving the same thank you page (or introduce a delay in responding to the spam form, if you want to be vindictive - this may not keep them from overwhelming your server and it may even let them overwhelm you faster, by keeping more connections open longer. At that point, you need a hardware solution, a firewall on a load balancer setup).

There are a lot of resources out there about delaying server responses to slow down attackers, frequently in the form of brute-force password attempts. This IT Security question looks like a good starting point.

I had been thinking about updating this question for a while regarding the topic of computer vision and form submission. An article surfaced recently that pointed me to this blog post by Steve Hickson, a computer vision enthusiast. Snapchat (apparently some social media platform? I've never used it, feeling older every day...) launched a new captcha-like system where you have to identify pictures (cartoons, really) which contain a ghost. Steve proved that this doesn't verify squat about the submitter, because in typical fashion, computers are better and faster at identifying this simple type of image.

It's not hard to imagine extending a similar approach to other Captcha types. I did a search and found these links interesting as well:

Oh, and we'd hardly be complete without an obligatory XKCD comic.

Wow, thank you for the information. I have read up on ways to prevent bots and most suggest CAPTCHA's, but lately I've been reading people say CAPTCHA's are not going to be around in the near future. THis gives me information that I can research, thank you.

I wouldn't say they won't be around in the near future. In my opinion, they have enough of a downside that they're falling out of favor for widespread use. There are plenty of stories (or rants) of Captchas making it way harder for legitimate users and not even stopping 100% of bot traffic. For senstivite applications, a level of hard-ness is acceptable, but if it's a small application or especially one where you benefit more than the user from them completing your form (e.g. feedback, or a business model with heavy competition), Captchas can cause you more problems than they solve.

In case of the register form, should I apply this measure too? Patrick M, check my profile, please.

In general, yes, you want to protect every input form with some kind of human detection. Registration forms typically require email verification; not for bot detection but to verify that you have some way to contact the user. If it's registration for an email service, well, check out what gmail does when you create a new account (and they have spam detection built into the sending protocol). If it's registration for a public forum, then absolutely use as much bot detection as you can, because (in my experience) that attracts the most bots looking for easy ways to spam.

I am not a lawyer and this is not legal advice. You may still violate any number of laws for user protections and privacy even if you provide the utmost bot detection.

javascript - Preventing bot form submission - Stack Overflow

javascript jquery forms security spam
Rectangle 27 3

Your code would not prevent bot submission but its not because of how your code is. The typical bot out there will more likely do an external/automated POST request to the URL (action attribute). The typical bots aren't rendering HTML, CSS, or JavaScript. They are reading the HTML and acting upon them, so any client logic will not be executed. For example, CURLing a URL will get the markup without loading or evaluating any JavaScript. One could create a simple script that looks for <form> and then does a CURL POST to that URL with the matching keys.

With that in mind, a server-side solution to prevent bot submission is necessary. Captcha + CSRF should be suffice. (http://en.wikipedia.org/wiki/Cross-site_request_forgery)

Than you for the information. I never realizes how sophisticated bots can actually be. My thought was, if the user has to interact the bot will not be able to perform it's job. I didn't realize bots can read Javascript and determine the PHP page. Would something like a token work to mitigate false posts?

javascript - Preventing bot form submission - Stack Overflow

javascript jquery forms security spam
Rectangle 27 3

Your code would not prevent bot submission but its not because of how your code is. The typical bot out there will more likely do an external/automated POST request to the URL (action attribute). The typical bots aren't rendering HTML, CSS, or JavaScript. They are reading the HTML and acting upon them, so any client logic will not be executed. For example, CURLing a URL will get the markup without loading or evaluating any JavaScript. One could create a simple script that looks for <form> and then does a CURL POST to that URL with the matching keys.

With that in mind, a server-side solution to prevent bot submission is necessary. Captcha + CSRF should be suffice. (http://en.wikipedia.org/wiki/Cross-site_request_forgery)

Than you for the information. I never realizes how sophisticated bots can actually be. My thought was, if the user has to interact the bot will not be able to perform it's job. I didn't realize bots can read Javascript and determine the PHP page. Would something like a token work to mitigate false posts?

javascript - Preventing bot form submission - Stack Overflow

javascript jquery forms security spam
Rectangle 27 2

eBay uses a pretty tricky method for logging in. It's a combination of cookies, hidden fields and a javascript redirect after successful login.

class Curl {
    private $ch;
    private $cookie_path;
    private $agent;

    // userId will be used later to keep multiple users logged
    // into ebay site at one time.
    public function __construct($userId) {
        $this->cookie_path = dirname(realpath(basename($_SERVER['PHP_SELF']))).'/cookies/' . $userId . '.txt';
        $this->agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)";
    }

    private function init() {
        $this->ch = curl_init();
    }

    private function close() {
        curl_close ($this->ch);
    }

    // Set cURL options
    private function setOptions($submit_url) {
        $headers[] = "Accept: */*";
        $headers[] = "Connection: Keep-Alive";
        curl_setopt($this->ch, CURLOPT_URL, $submit_url);
        curl_setopt($this->ch, CURLOPT_USERAGENT, $this->agent); 
        curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, 1);  
        curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false);
        curl_setopt($this->ch, CURLOPT_HTTPHEADER,  $headers);
        curl_setopt($this->ch, CURLOPT_COOKIEFILE, $this->cookie_path);         
        curl_setopt($this->ch, CURLOPT_COOKIEJAR, $this->cookie_path);
    }

    // Grab initial cookie data
    public function curl_cookie_set($submit_url) {
        $this->init();
        $this->setOptions($submit_url);
        curl_exec ($this->ch);
        echo curl_error($this->ch);
    }  

    // Grab hidden fields
    public function get_form_fields($submit_url) {
        curl_setopt($this->ch, CURLOPT_URL, $submit_url);
        $result = curl_exec ($this->ch);
        echo curl_error($this->ch);
        return $this->getFormFields($result);
    }

    // Send login data
    public function curl_post_request($referer, $submit_url, $data) {
        $post = http_build_query($data);
        curl_setopt($this->ch, CURLOPT_URL, $submit_url);
        curl_setopt($this->ch, CURLOPT_POST, 1);  
        curl_setopt($this->ch, CURLOPT_POSTFIELDS, $post);  
        curl_setopt($this->ch, CURLOPT_REFERER, $referer);
        $result =  curl_exec ($this->ch);
        echo curl_error($this->ch);
        $this->close();
        return $result;
    }    

    // Show the logged in "My eBay" or any other page
    public function show_page( $submit_url) {
        curl_setopt($this->ch, CURLOPT_URL, $submit_url);
        $result =  curl_exec ($this->ch);
        echo curl_error($this->ch);
        return $result;
    }

    // Used to parse out form
    private function getFormFields($data) {
        if (preg_match('/(<form name="SignInForm".*?<\/form>)/is', $data, $matches)) {
            $inputs = $this->getInputs($matches[1]);
            return $inputs;
        } else {
            die('Form not found.');
        }
    }

    // Used to parse out hidden field names and values
    private function getInputs($form) {
        $inputs = array();
        $elements = preg_match_all('/(<input[^>]+>)/is', $form, $matches);

        if ($elements > 0) {
            for($i = 0; $i < $elements; $i++) {
                $el = preg_replace('/\s{2,}/', ' ', $matches[1][$i]);

                if (preg_match('/name=(?:["\'])?([^"\'\s]*)/i', $el, $name)) {
                    $name  = $name[1];
                    $value = '';

                    if (preg_match('/value=(?:["\'])?([^"\'\s]*)/i', $el, $value)) {
                        $value = $value[1];
                    }

                    $inputs[$name] = $value;
                }
            }
        }

        return $inputs;
    }

    // Destroy cookie and close curl.
    public function curl_clean() {
        // cleans and closes the curl connection
        if (file_exists($this->cookie_path)) { 
            unlink($this->cookie_path); 
        }
        if ($this->ch != '') { 
            curl_close ($this->ch);
        }
    }    
}
$curl = new Curl(md5(1));   //(md5($_SESSION['userId']));
$referer = 'http://ebay.com';
$formPage = 'http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn';

// Grab cookies from main page, ebay.com
$curl->curl_cookie_set($referer);

// Grab the hidden form fields and then set UsingSSL = 0
// Login with credentials and hidden fields
$data = $curl->get_form_fields($formPage);
$data['userid'] = "";
$data['pass'] = "";
$data['UsingSSL'] = '0';

// Post data to login page. Don't echo this result, there's a
// javascript redirect. Just do this to save login cookies
$formLogin = "https://signin.ebay.com/ws/eBayISAPI.dll?co_partnerId=2&amp;siteid=3&amp;UsingSSL=0";
$curl->curl_post_request($referer, $formLogin, $data);

// Use login cookies to load the "My eBay" page, viola, you're logged in.
$result = $curl->show_page('http://my.ebay.com/ws/eBayISAPI.dll?MyeBay');

// take out Javascript so it won't redirect to actualy ebay site
echo str_replace('<script', '<', $result);

you better hide your username and password credintials

php - cURL login into ebay.co.uk - Stack Overflow

php curl ebay
Rectangle 27 1

Since t.co uses HTML redirection through the use of JavaScript and/or a <meta> redirect we need to grab it's contents first. Then extract the bit.ly URL from it to perform a HTTP header request to get the final location. This method does not rely on cURL to be enabled on server and uses all native PHP5 functions:

Tested and working!

function large_url($url) 
{
    $data = file_get_contents($url); // t.co uses HTML redirection
    $url = strtok(strstr($data, 'http://bit.ly/'), '"'); // grab bit.ly URL

    stream_context_set_default(array('http' => array('method' => 'HEAD')));
    $headers = get_headers($url, 1); // get HTTP headers

    return (isset($headers['Location'])) // check if Location header set
        ? $headers['Location'] // return Location header value
        : $url; // return bit.ly URL instead
}

// DEMO
$url = 'http://t.co/dd4b3kOz';
echo large_url($url);
$url=http://t.co/dd4b3kOz;echo large_url($url);
http://bit.ly/IRnYVz
http://changeordie.therepublik.net/?p=371#proliferation

@ChetanaKestikar So you are trying to expand a URL that is shortened with bit.ly and then further shortened with Twitter? TinyURL is not being used at all. Ok, before I modify the code please tell me if you are sure these are the only two URL shortening services applied to the URL.

yeah ... these are the only ones...

t.co
<noscript><META http-equiv="refresh" content="0;URL=http://bit.ly/IRnYVz"></noscript><script>location.replace("http:\/\/bit.ly\/IRnYVz")</script>

Okay, i did not know that...Yeah sure take ur time....

php - Get Final URL From Double Shortened URL (t.co -> bit.ly -> final...

php url curl
Rectangle 27 4

Maybe what you can do is write a server-side wrapper function. That way your javascript only does a single asynchronous call to your own web server. Then your web server uses curl (or urllib, etc.) to interact with the remote API.

@PatrikAkerstrand, urllib on server - much harder to read than nested callbacks on client. =)

javascript - How to chain ajax requests? - Stack Overflow

javascript jquery asynchronous
Rectangle 27 13

cURL will only get you the markup of the page. It won't load any additional resources or process the page. You probably want to look at something like PhantomJS for this. PhantomJS is a headless WebKit browser. It has its own API that lets you "script" behavior. So you can tell PhantomJS to load the page and dump out the data you need.

Thanks for your answer. I will need to run this as JavaScript and then save the dump to a PHP variable using the exec command. Is that correct?

You don't have to, actually. You can run it directly from the command line. But if you are using this to display it on a website, then yes you can use exec from PHP.

How to get webcontent that is loaded by JavaScript using cURL? - Stack...

javascript curl
Rectangle 27 1

To do what http://www.hidemyass.com/ is doing is impossible with just javascript, because javascript is client side. This site likely acts as a server proxy... almost, which means it requires server side communication (probably). (Likely the reason they're asking for money because they handle the traffic) You will likely need AJAX to do this which requires a server with some ability to handle and access other servers without restriction. Any free server will not do this easily and often have a policy on servers that they track communications to. So you are likely to have to pay somebody, if you aren't already. this is cheep if it's just you but the more traffic the more expensive.

That said, you can see if what sandbox mode offers is right for you.

<iframe sandbox><iframe>
  • (no value) - - - - - - - - - - - - - - - -Applies all restrictions
  • allow-forms - - - - - - - - - - - - - - Re-enables form submission
  • allow-pointer-lock - - - - - - - - - - Re-enables APIs
  • allow-popups - - - - - - - - - - - - - Re-enables popups
  • allow-same-origin - - - - - - - - - -Allows the iframe content to be treated as being from the same origin
  • allow-scripts - - - - - - - - - - - - - Re-enables scripts
  • allow-top-navigation - - - - - - - - Allows the iframe content to navigate its top-level browsing context
<iframe sandbox="allow-forms"><iframe>

If that does not satisfy, play with onclick and onchange events to stop users from interacting. with clickable elements or moving to other pages.

<iframe onchange="function(){\
  document.getElementsByTagName("iframe"[0].src =\
  "http://facebook.com");\
}"></iframe>

or some kind of onclick prevent default.

It's not safe in the first place so why stop there.

Access the iframe.contentDocument and search and remove all <a...</a> tags or just the ones you want. there is some kind of remove child method you can look up to do this. if you want to be more savvy you can compare href's an see if it belongs to a foreign site.

javascript - how to force link from iframe to be opened in the parent ...

javascript html iframe curl
Rectangle 27 1

The CSS (and probably some Javascript as well) don't load because they're using absolute or relative paths that have no meaning on your domain.

You're going to either need to find all these links in the cURL response and replace them with valid URLs on the domain (and probably rewrite regular HTML links and other assets).

Your best bet is probably to configure Apache to act as a reverse proxy so it'll do all this for you. See ProxyPass.

I didn't have access to Apache because it is on shared hosting, can you give me some tips how I can do this in other way?

php - cURL get page content like iframe - Stack Overflow

php curl iframe file-get-contents
Rectangle 27 1

You are looking for a div with and ID of events. That exists in the page but it is indeed an empty element, at least on page load. It gets filled using ajax, but you will not get that information when you use cURL to get the page. Or any other method that does not parse the page and execute the javascript.

However, you are in luck. They are making an ajax call to:

http://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard?calendartype=blacklist&dates=20150410

And you can easily do the same.

It will get you the information as a json string but that is easy to parse using json_decode in php. Afterwards you will have a nested object or array and you can display the data as you please.

Please note: I don't know if you are allowed to do that so how you use this, is up to you. You could try and search on their site if they offer their API publicly and what the conditions are.

but I believe that there should be some kind of authentication to access their api.

@NuttyProgrammer Nope, I tried it from another browser before I posted this and without opening the web-page there first, I get the complete json. They could be doing something that is ip based but you would have to try that.

@NuttyProgrammer Just tried it from my phone on another internet connection and it opens without any form of authentication.

javascript - How to target an inner article tag using PHP Simple HTML ...

javascript php html web-scraping simple-html-dom
Rectangle 27 1

You are looking for a div with and ID of events. That exists in the page but it is indeed an empty element, at least on page load. It gets filled using ajax, but you will not get that information when you use cURL to get the page. Or any other method that does not parse the page and execute the javascript.

However, you are in luck. They are making an ajax call to:

http://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard?calendartype=blacklist&dates=20150410

And you can easily do the same.

It will get you the information as a json string but that is easy to parse using json_decode in php. Afterwards you will have a nested object or array and you can display the data as you please.

Please note: I don't know if you are allowed to do that so how you use this, is up to you. You could try and search on their site if they offer their API publicly and what the conditions are.

but I believe that there should be some kind of authentication to access their api.

@NuttyProgrammer Nope, I tried it from another browser before I posted this and without opening the web-page there first, I get the complete json. They could be doing something that is ip based but you would have to try that.

@NuttyProgrammer Just tried it from my phone on another internet connection and it opens without any form of authentication.

javascript - How to target an inner article tag using PHP Simple HTML ...

javascript php html web-scraping simple-html-dom
Rectangle 27 45

For the broader problem of passing a JSON-encoded string in PHP (e.g., through cURL), using the JSON_HEX_APOS option is the cleanest way to solve this problem. This would solve your problem as well, although the previous answers are correct that you don't need to call parseJSON, and the JavaScript object is the same without calling parseJSON on $data.

json_encode($var)
json_encode($var, JSON_HEX_APOS)

Here's an example of the correctly encoded data being parsed by jQuery: http://jsfiddle.net/SuttX/

For further reading, here's an example from the PHP.net json_encode manual entry Example #2:

$a = array('<foo>',"'bar'",'"baz"','&blong&', "\xc3\xa9");

echo "Apos: ",    json_encode($a, JSON_HEX_APOS), "\n";
Apos: ["<foo>","\u0027bar\u0027","\"baz\"","&blong&","\u00e9"]

A full list of JSON constants can be found here: PHP.net JSON constants

Thanks you! This solution works very well for me!

Awesome and simple solution! Saved my life 2 years later. Thanks!!

php json_encode & jquery parseJSON single quote issue - Stack Overflow

php jquery
Rectangle 27 146

There is a method that is designed for this. Check out process.hrtime(); .

var start = process.hrtime();

var elapsed_time = function(note){
    var precision = 3; // 3 decimal places
    var elapsed = process.hrtime(start)[1] / 1000000; // divide by a million to get nano to milli
    console.log(process.hrtime(start)[0] + " s, " + elapsed.toFixed(precision) + " ms - " + note); // print message + time
    start = process.hrtime(); // reset the timer
}

Then I use it to see how long functions take. Here's a basic example that prints the contents of a text file called "output.txt":

var debug = true;
http.createServer(function(request, response) {

    if(debug) console.log("----------------------------------");
    if(debug) elapsed_time("recieved request");

    var send_html = function(err, contents) {
        if(debug) elapsed_time("start send_html()");
        response.writeHead(200, {'Content-Type': 'text/html' } );
        response.end(contents);
        if(debug) elapsed_time("end send_html()");
    }

    if(debug) elapsed_time("start readFile()");
    fs.readFile('output.txt', send_html);
    if(debug) elapsed_time("end readFile()");

}).listen(8080);

Here's a quick test you can run in a terminal (BASH shell):

for i in {1..100}; do echo $i; curl http://localhost:8080/; done

it that superior to the console.time solution in any way?

Yep, it's a lot more precise and you can store the result in a variable

node.js - How to measure execution time of javascript code with callba...

javascript node.js profiling
Rectangle 27 146

There is a method that is designed for this. Check out process.hrtime(); .

var start = process.hrtime();

var elapsed_time = function(note){
    var precision = 3; // 3 decimal places
    var elapsed = process.hrtime(start)[1] / 1000000; // divide by a million to get nano to milli
    console.log(process.hrtime(start)[0] + " s, " + elapsed.toFixed(precision) + " ms - " + note); // print message + time
    start = process.hrtime(); // reset the timer
}

Then I use it to see how long functions take. Here's a basic example that prints the contents of a text file called "output.txt":

var debug = true;
http.createServer(function(request, response) {

    if(debug) console.log("----------------------------------");
    if(debug) elapsed_time("recieved request");

    var send_html = function(err, contents) {
        if(debug) elapsed_time("start send_html()");
        response.writeHead(200, {'Content-Type': 'text/html' } );
        response.end(contents);
        if(debug) elapsed_time("end send_html()");
    }

    if(debug) elapsed_time("start readFile()");
    fs.readFile('output.txt', send_html);
    if(debug) elapsed_time("end readFile()");

}).listen(8080);

Here's a quick test you can run in a terminal (BASH shell):

for i in {1..100}; do echo $i; curl http://localhost:8080/; done

it that superior to the console.time solution in any way?

Yep, it's a lot more precise and you can store the result in a variable

node.js - How to measure execution time of javascript code with callba...

javascript node.js profiling
Rectangle 27 2

Unfortunately, unless you can grab the HTML via PHP or some other scripting language (using cURL in PHP possibly) which would then pass it back to your JavaScript application from the same domain, this is not possible. Please refer to the Same Origin Policy.

Cross-domain iframe jquery selector - Stack Overflow

jquery iframe