ajax.Rmd

# AJAX Requests {#ajax}
JavaScript allows you to dynamically define the content of a web page, generating the DOM at runtime rather than in `.html` source files. One of the primary reasons we would want to dynamically produce a DOM is if the web page's content is based on some **data** that may change over time: for example, the kind of data that is available through a [**Web API**](https://info201.github.io/apis.html#what-is-a-web-api). By using JavaScript to render the DOM, you can quickly produce large amounts of HTML needed to display large data sets, sure that you have up-to-date data each time the page loads, and even _automatically refresh the page content_ without requiring the user to reload!

This chapter describes how to use JavaScript to dynamically send HTTP Requests to download data (without reloading the page!), as well as how to perform the **asynchronous programming** needed when working with web requests and other time-consuming operations.

<p class="alert alert-info">Note that this lecture assumes that you have a basic familiarity with RESTful Web APIS, including how to read and access their endpoints. For a review of some of the terminology used in APIS and RESTful requests, see the [INFO 201 course reader](https://info201.github.io/apis.html#restful-requests).</p>


## AJAX
As discussed in [Chapter 2](#http-requests-and-servers), you download data from the Internet by sending an **HTTP Request** and then processing the **response**. In everyday usage, HTTP Requests are normally sent by the _browser_ when the user enters a URL or clicks on a link. By default, if you wanted to download new data, you'd need to have the browser send a new request, loading a new page (or reloading the current page) in order to show that result.

To make modern dynamic web pages that display new data without needing to refresh the browser, we use a technique to send HTTP Requests _from JavaScript code_ rather than from the browser. This allows us to "by-pass" the browser and get new data (and change the webpage) without reloading it! This technique is referred to as **AJAX** (**A**synchronous **J**avaScript **A**nd **X**ML)&mdash;we write code that sends an "request with AJAX" or an "AJAX request".

<p class="alert alert-info">_Fun fact_: The technology used to send AJAX requests was originally developed by Microsoft in the late 90s to support their fledgling web version of the Outlook email/calendar app. The JavaScript functions used to send these requests were included in Internet Explorer as a _non-standard_ feature&mdash;an example of a browser adding new functionality that it thinks will be useful but that doesn't work on other platforms. However, AJAX quickly gained popularity (particularly when Google showed off what you could do with it via Gmail and Google Maps), and has since become a standard that is now supported by all browsers. This is how standards come into existence!</p>

### XML and JSON {-}
AJAX is called "AJA**X**" because it was originally designed to request data in XML format. **XML** (E**X**tensible **M**arkup **L**anguage) is a markup language (like HTML) that is used to encode meaning in content in a format that is both human _and_ computer readable. The syntax for XML is _exactly_ the same as HTML: in fact, HTML can be seen as a "subset" of the language. You can think of XML as "HTML, but you get to make up your own element names!"

```xml
<!-- Some XML encoding information about a person -->
<person>
   <firstName>Alice</firstName>
   <lastName>Smith</lastName>
   <favorites>
      <music>jazz</music>
      <food>pizza</food>
      <numbers>
          <item>12</item>
          <item>42</item>
      </numbers>
   </favorites>
</person>
```

- The XML language does not define any particular tags the way HTML does; instead it is up to individual applications to determine what tags it will recognize and interpret (and what tags it would see as gibberish)&mdash;what is referred to as a [XML Schema](https://en.wikipedia.org/wiki/XML_schema).

At the time AJAX was first developed, XML was the most common way of encoding generic data for transmission. And because XML is a tree of elements just like the DOM, similar methods could be used to navigate and extract information from the tree. However, XML is a very _verbose_ language: it requires a lot of characters to encode information (meaning that the amount of data being transferred is larger), and traversing an element tree requires a lot of code. As such, JavaScript developers (led by [Douglas Crockford](https://en.wikipedia.org/wiki/Douglas_Crockford)) developed an alternative language called **JSON** (**J**ava**S**cript **O**bject **N**otation) that is more compact than XML and can be _directly_ parsed into JavaScript objects and arrays:

```json
{
  "firstName": "Alice",
  "lastName": "Smith",
  "favorites": {
    "music": "jazz",
    "food": "pizza",
    "numbers": [12, 42]
  }
}
```

JSON format uses a syntax that is almost identical to that for defining Object literals in JavaScript, with a few key differences:

<div class="list-condensed">
- JSON always defines an Object `{}` at the "top level".
- JSON object **keys** (which must be strings) _must_ be written in double-quotes.
- JSON **values** can only be strings, numbers, booleans (`true` or `false`), arrays (`[]`), or other objects. You cannot include a function in JSON.
- JSON objects and arrays can't have trailing commas or other extraneous symbols&mdash;no comments!
</div>

The JavaScript language provides a global object `JSON` (like the global `Math` object) that can be used to convert from encoded _strings_ of JSON content (e.g., the above code block as a single string variable `'{"firstName":"Alice"}'`) to JavaScript objects, and vice versa:

```js
//convert from Object to encoded String
let personObj = {firstName:"Alice", lastName:"Smith", id:12} //JavaScript object
let personString = JSON.stringify(personObj); //turn object into JSON string
console.log(personString); //=> '{"firstName":"Alice","lastName":"Smith","id":12}'
console.log(typeof personString); //=> 'string'

//convert from encoded String to Object
let favoritesString = '{"music":"jazz", "numbers":[12,42]}'; //a string, not an object!
let favoritesObj = JSON.parse(favoritesString); //turn JSON string into object
console.log(favoritesObj); //=> { music: 'jazz', numbers: [ 12, 42 ] }
console.log(typeof favoritesObj); //=> 'object'
```

- Note that if your JSON string is not properly formatted (e.g., you're missing a quote), the `JSON.parse()` function will throw a `SyntaxError`. The exact error in the JSON string can be hard to find; [online tools](https://jsonformatter.curiousconcept.com/) can help show the problem.

JSON has replaced XML as the encoding of choice for working with AJAX requests&mdash;however, the technique is still referred as "AJAX" ("AJA*J*" isn't as easy to say!)


## Fetching Data
AJAX support is built into browsers through the included `XMLHttpRequest` global variable (the "xml http thing"). This object provides functions that allow you to send an HTTP request to the server, but the object's API is **really complex to use**:

<details>
<summary>An example `XMLHTTPRequest`</summary>
```js
//create a new XMLHttpRequest object
let request = new XMLHttpRequest();

//configure it to do an HTTP GET request for some URL
request.open('GET', 'https://domain.com/data', true);

//add a listener for the "load" event (when the data has been downloaded)
request.addEventListener('load', function() {
    if (request.status >= 200 && request.status < 400) { //check response status
        let data = JSON.parse(request.responseText);
        console.log(data); //do something with the data
    }
});

//listen for "error" events if there was a network error
request.addEventListener('error', function() {
    //handle error...
})

//finally, send the request to the server!
request.send();
```
</details>

Instead of needing to understand all that code, developers tended to use functions from external libraries such as jQuery's [`$.getJSON()`](http://api.jquery.com/jQuery.getJSON/) or [`$.ajax()`](http://api.jquery.com/jQuery.ajax/):

```js
$.getJSON('https://domain.com/data', function(data) {
    //`data` is the already-parsed JSON data
    console.log(data); //do something with the data
});
```

But this requires including the jQuery library in your page, and [since the need for jQuery is rapidly going away](http://youmightnotneedjquery.com/), other options are now _built in_ to modern browsers. In particular, we will utilize the [**`fetch()`**](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) API to easily send AJAX requests for data!

<div class="alert alert-warning">
`fetch()` is an recent standard, so that it is not supported by older browsers (e.g., [Internet Explorer](http://caniuse.com/#feat=fetch). However, we can still use `fetch()` with these browsers by including a [**polyfill**](https://en.wikipedia.org/wiki/Polyfill)&mdash;an external library that replicates an existing API in platforms that don't support it! The [`fetch()` polyfill](https://github.com/github/fetch) will provide a `fetch()` function to browsers that don't provide it (leaving other browsers unchanged) that uses the existing `XMLHttpRequest` without you needing to interact with that object.</p>

It's easiest to just load the polyfill from a [CDN](https://cdnjs.com/libraries/fetch):

```html
<!-- put this BEFORE your own script! -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/fetch/2.0.3/fetch.min.js"></script>
```
</div>

The `fetch()` function makes it easy to send a request: simply call it and pass in the URL of the data you wish to download:

```js
fetch('https://domain.com/data');
```

<!-- //same origin policy & live-server?? Necessary?? -->

<p class="alert alert-warning">In some browsers, you will not be able to send an AJAX request when the web page is loading via the `file://` protocol (e.g., by double-clicking on the `.html` file). This is a security feature to keep you from accidentally running an HTML file that contains malicious JavaScript that will download a virus onto your computer. Instead, you should run a local web server such as `live-server` for testing AJAX requests.</p>

<p class="alert alert-success">Remember to always use a **relative path** when specifying what file or API you wish to fetch. Moreover, paths are always relative to the `.html` file that the user is currently viewing, not to the `.js` file that contains the `fetch()` call. So be careful if your JavaScript is in a different folder; the path needs to be relative to the HTML (which should be in the root of your project).</p> 

## Asynchronous Programming
However, the `fetch()` function does **NOT** directly return the data you want to download! Downloading data off the internet can take a long time: the network connection may be slow and the amount of data to download may be quite large (metadata for the latest 100 tweets from Twitter involves almost 500k of JSON content). Because fetching data may take time, AJAX requests are made **asynchronously** (that's the "A" in AJAX)&mdash;the download will occur _at the same time_ that the rest of the code is being executed. Thus the download and the remaining script will _not_ be synchronized; they will be "asynchronous".

```js
console.log('About to send request'); //statement 1

//send request for data to the url
fetch(url);  //statement 2

console.log('Sent request'); //statement 3

//The data is actually received sometime later,
//when the JS interpreter is down here!
```

- In the above example, the JS interpreter will execute statement 1, then statement 2 (the `fetch()` call). It will then precede _immediately_ to statement 3 (the second `console.log`), without waiting for the request to finish! The download will continue to occur in the background, and will finish at some point later in the program&mdash;though we don't know exactly when.

- It is best to think of `fetch()` as a function that will just "_start_ to download data", not one that actually downloads data!

Because `fetch()` is an **asynchronous function** (it's code is run asynchronously), it returns what is called a [**Promise**](https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/Promise.jsm/Promise). A Promise is object that holds a value which may not be available yet&mdash;you can think of a Promise as like a placeholder where the result of the asynchronous function call will eventually be stored (it is a "promise" to eventually have some data, though that promise may be kept or broken!).

- Promises are the modern way of handling asynchronous functions, but as part of the ES6 standard they are [not yet available to all browsers](http://caniuse.com/#feat=promises) (specifically: Internet Explorer). So you'll need to include [_another polyfill_](https://github.com/stefanpenner/es6-promise) to support IE. This is also available from a [CDN](https://cdnjs.com/libraries/es6-promise).

Promises have three possible states: _pending_ (the data is downloading), _fulfilled_ (the data has finished downloading), or _rejected_ (the data failed to download and the promise was "broken"). We are able to specify _callback functions_ (similar to event listeners) that occur when a pending Promise is either successfully fulfilled or has been rejected. The "on success" callback function is specified by calling the **`.then()`** function on the Promise object, and passing the "on success callback" as a parameter:

```js
function onSuccessCallback(response) { //what to do when we get the response
    console.log(response);
}

//When fulfilled, execute the callback function (which will be passed the response)
let promise = fetch(url);
promise.then(onSuccessCallback);

//It is much more common to use anonymous variables/callbacks:
fetch(url).then(function(response) {
    console.log(response);
});
```

The "on success" callback will be passed a single parameter: the **data value** that the Promise was made for (e.g., the data that will eventually be downloaded from `fetch()`). So when the callback is executed, you will have access to the data! For example, when the `fetch()` Promise is fulfilled, it will pass an object representing the _response_ to the HTTP Request:

```js
let promise = fetch(url);
promise.then(function(response){
    console.log( response.url ); //a string of where the request was sent
    console.log( response.status ); //the HTTP status code (e.g., 200, 404)
});
```

This response object does have a `body` property that represents the "body" (data content) of the HTTP response. However, that body stored as a "stream" of 0s and 1s, not as a JavaScript object (or even a string you can `JSON.parse()`)! In order to get the body into a format you can use, you will need to "encode" it into a JavaScript object by calling the **`.json()`** method on it.

- There is also an equivalent `.text()` method to encode a response body into plain text.

### Chaining Promises {-}
**But there's a catch**: the "encoding" process performed by the `.json()` might take some time (particularly for a large amount of data). So instead of blocking (pausing) the rest of your program while that encoding occurs, the `.json()` method returns _another Promise_ as a placeholder for when the encoded body is available! So you will then need to specify a `.then()` callback for _that_ Promise as well.

However, a Promise's `.then()` function has a neat property that makes this easy to do. Calling the `.then()` function on a Promise returns a new Promise as a placeholder for any data produced by the `.then()` function. This promised data will be whatever value is _returned_ by the "on success" callback function. This allows you to in effect "chain" `.then()` calls together, each of which can perform some kind of transformation on the data:

```js
function makeQuestion(dataString) { //a function to make a string a question
    return dataString + '???';
}

//image a hypothetical asynchronous function `getAsyncString`
//it returns a Promise (placeholder) for a string load from a given source
let originalPromise = getAsyncString(myDataSource);

//when the original promise is fulfilled, call `makeQuestion` on it
//`questionPromise` will be a placeholder for that transformed data
let questionPromise = originalPromise.then(makeQuestion);

//when the `questionPromise` is fulfilled, call an anonymous callback on it
//the callback will be passed the transformed ("question") data
questionPromise.then(function(data){
    console.log(data); //data will be a question!
})
```

More commonly, we use _anonymous variables_ for subsequent promises, allowing you to chain them together in a way that almost reads like English!

```js
getAsyncString()
    .then(makeQuestion)
    .then(function(data){
        console.log(data);
    });
```

But wait there's more! `.then()` also has a special feature where if the "on success" callback function returns a _Promise_ (rather than another kind of value), then the "outer" promise will take on the state of that new returned Promise. This means that you can just _return_ a Promise from inside a `.then()` callback, and that Promise will be the subject of the subsequent `.then()` call:

```js
let outerPromise = getAsyncString(myFirstSource).then(function(firstData){
    //do something with `firstData`

    let newPromise = getAsyncString(mySecondSource); //a second async call!
    return newPromise; //return the promise.
}); //`outerPromise` now takes on the state and data of `newPromise`

outerPromise.then(function(secondData){
    //do something with `secondData`, the data downloaded from `mySecondSource`
});
```

Going back to `fetch()` to bring it all together: since the `.json()` encoding function returns a Promise, you can simply _return_ that Promise from the `.then()` callback in order to make it available to subsequent `.then()` calls!

```js
fetch(url)  //start the download
    .then(function(response) {  //when done downloading
        let dataPromise = response.json();  //start encoding into an object
        return dataPromise;  //hand this Promise up
    })
    .then(function(data) {  //when done encoding
        //do something with the data!!
        console.log(data); //will now be encoded as a JavaScript object!
    });
```

This code example will allow you to download data and encode it into a plain old JavaScript object that you can work with.

### Handling Errors {-}
When downloading data from the internet, it is always possible that the HTTP request may fail. The request may be sent to the wrong URL, the client computer may be having connection problems, or the receiving server may be having problems.

In order to deal with inevitable errors, Promises provide a **`.catch()`** method that is used to specify a callback that should occur if the Promise is _rejected_ (an error occurs). This callback function will be passed an [Error object](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) that contains details about the error.

```js
fetch(url)
    .then(function(response) {  //when done downloading
        return response.json();  //second promise is anonymous
    })
    .then(function(data) {  //when done encoding
        //do something with the data!!
        console.log(data); //will now be encoded as a JavaScript object!
    })
    .catch(function(err) {
        //do something with the error
        console.error(err);  //e.g., show in the console
    });

```

- Importantly, the `.catch()` method will "catch" errors from _all previous_ Promises in a `.then()` chain! This means that the above `.catch()` will show both errors in the downloading (`.fetch()`), and errors in the body encoding (`.json()`).

- You will almost always want to show the error to the user in some way, such as by creating an [alert](http://getbootstrap.com/docs/4.0/components/alerts/) element in the DOM.

- The `.catch()` function _also_ returns a Promise, so you can continue to chain `.then()` calls after it. These later callbacks will only be executed if no previous Promise has been rejected (that is, there haven't been any errors yet).

**Important**: a Promise will only be rejected if there is an actual "Error" in sending the request. If the server replies with a [401](https://httpstatuses.com/401) error (e.g., you didn't have permission to access the resource) or just the message "invalid API key", that won't be handled by `.catch()`. From JavaScript's perspective, the request went through perfectly&mdash;it's not fetch's fault that the data you asked for wasn't what you actually wanted!

- You can use the `response.status` and `response.ok` properties to check the status of the HTTP response.

As such, you will want to make sure to handle things like bad responses or unexpected response bodies, both in testing your application (to make sure the request is sent to the correct URL) and when handling any user input.


<!-- TODO
## async/await
//this lets you write asynchronous code in a more "synchronous" style, by specifying lines of code that will only execute after the promise is resolved (fulfilled or rejected).

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
https://hackernoon.com/6-reasons-why-javascripts-async-await-blows-promises-away-tutorial-c7ec10518dd9
-->


<!--
//forms go here... or just walk through making one in exercise!
    //form: `method` and `action`
    //input elements
    //labels
    //aria
    //preventing default
-->

### Other Data Formats {-}
The above usage of `fetch()` works well for data formatted in JSON (which is the most common format for web-accessible data). However, you may wish to dynamically load data that is presented in a different format, such as plain text or as comma-separated values (CSV). There are a few ways to support this:

For downloading **plain text** formatted data, you can use the `fetch()` method as above, but instead of calling `.json()` on the response to encode it as a JavaScript object, you can call the `.text()` method to encode it as a basic string:

```js
fetch(url)  //start the download
    .then(function(response) {  //when done downloading
        let dataPromise = response.text();  //start encoding into a String
        return dataPromise;  //hand this Promise up
    })
    .then(function(text) {  //when done encoding
        //do something with the text data!!
        console.log(text); //will now be encoded as a JavaScript string!
    });
```

Encoding as text will support _any_ plaintext formated data&mdash;whether from a `.txt` file, a `.csv` file, or even a `.json` or `.js` file! In fact, if you encode JSON data as plain text using the `.text()` method, you could then explicitly parse that into a JavaScript Object by using the built-in [`JSON.parse()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse) function described above!

While the `.text()` method will you encode data into a JavaScript String, that String will often have a particular format that needs to be **parsed** (interpreted) in order to make the data useful. For example, a String in CSV format is not very useful on its own; you would need to parse it into an array of objects (where each object represents a row) to analyze the data.

It is possible ([but complex](https://stackoverflow.com/a/8497474)) to do this parsing using built-in String functions (assuming your CSV data matches official standards), but in general it's easier and more effective to use an external library to do this parsing. One of the best libraries for doing this work is [**`d3.dsv()`**](https://github.com/d3/d3-fetch), a component of the [d3](https://d3js.org/) visualization library. `d3.fetch()` provides convenience wrapper functions for `fetch()` that also perform effective data parsing.

In order to use `d3.fetch()`, you will need to load it as an external library:

```html
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-fetch.v1.min.js"></script>
```

(More details coming soon...)
<!-- (the `d3-fetch` )



```
'{"name":"Ada Smith", "age": }'
```


For formatted plain text (such as JSON or CSV data), you will need -->






## Resources {-}
<div class="list-condensed">
- [An Introduction to AJAX for Front-End Developers (tuts+)](https://webdesign.tutsplus.com/tutorials/an-introduction-to-ajax-for-front-end-designers--cms-25099)
- [An Introduction to `fetch()` (Google)](https://developers.google.com/web/updates/2015/03/introduction-to-fetch)
- [JavaScript Promises: an Introduction (Google)](https://developers.google.com/web/fundamentals/primers/promises)
</div>

<!-- http://papaparse.com/ <- csv parsing -->