LIS768_Lab5_2023.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5: Acquiring data\n",
    "## Relevant Python basics\n",
    "In this lab, we'll use a few things we've covered so far in this class. \n",
    "- We'll *import* a python library called `requests`. Depending on the type of files you intend to work with, you may import a few other libraries as well.\n",
    "- We'll use several pre-built *functions* from the standard Python library, a few from `requests`, and, depending on your data formats, a few others as well.\n",
    "- We'll store some things as *variables*.\n",
    "- We'll also use *lists* and *loops* to automate some aspects of this task.\n",
    "\n",
    "**You may find previous Python labs handy for this lab.**\n",
    "\n",
    "## Part 1: Reading and writing files in Python\n",
    "Before we begin gathering data from online sources--scraping, downloading, or querying APIs--we'll need a few more Python techniques to allow us to create and interact with files on our computers. In this section, we will practice a couple basic techiques, which we can build on to handle additional file types.\n",
    "\n",
    "### File handling\n",
    "Regardless of whether you want to **read** or **write** files in Python, you'll want to familiarize yourself with the `open()` function. `open()` is part of the standard Python library and it allows you to interact with the file in various ways. \n",
    "\n",
    "#### If there's an `open()`, is there a `close()` function, too?\n",
    "Yep! There is and you should be sure to use it any time you've opened a file...unless(!) you use the little bit of shorthand that I have written into all our labs:\n",
    "\n",
    "`with open()`\n",
    "\n",
    "A detailed explanation of how this works is outside the scope of our class, but what you need to know is that this shorthand **will make sure that your file is closed when you're done using it.** For this reason, we're going to skip straight to using `with open()` instead of first practicing `open()` and `close()`. If you'd like to explore those further, there's a decent walkthrough on [W3 Schools](https://www.w3schools.com/python/python_file_handling.asp).\n",
    "\n",
    "### 👓 Reading and writing files ✏️\n",
    "Using our `with open()` technique, we can tell the computer essentially how we want it to open those files. We've got four methods to choose from:\n",
    "- `\"r\"` read: This is the default. It allows you to read the contents of the file, but does not allow you to change them in any way. It will return an error if no file by that name exists in the specified location.\n",
    "- `\"w\"` Write: This allows you to write to a file. Choosing this option will allow you to overwrite the contents of a file--be careful with this one. Importantly, it will also create a new file with the specified name if one does not exist.\n",
    "- `\"a\"` Append: This allows you to append--or add content to the end of--the file, but does not allow you to overwrite the contents of the file. If a file by the specified name doesn't exist in this location on your computer, it will create one.\n",
    "- `\"x\"` Create: This will create a file if one doesn't already exist and it will return an error if one does.\n",
    "\n",
    "You can also tell the computer what mode the file should be handled in (this is a bit more technical):\n",
    "- `\"t\"` text is the default and it will treat the contents as text.\n",
    "- `\"b\"` binary is the other option for non-text file types. \n",
    "\n",
    "In the example below, I will demonstrate the syntax for using `with open()` to open the text file you downloaded alongside this lab.\n",
    "\n",
    "**Run the code cell below to open a sample file.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In the first line, we'll use the keyword 'with' and the `open()` function. \n",
    "#Inside the parenthesis, inside quotes, is the name of the file I want to open.\n",
    "#After that, is a keyword 'as', which lets us give an alias to the file we're opening.\n",
    "#The alias will let us call the file as we work with it.\n",
    "\n",
    "#In the second line, we read the contents of sampleText using the read() function and store them in a variable.\n",
    "#Note that everything related to reading the file is indented after the first line.\n",
    "#The last line of this block of code prints the values stored in 'content' to the screen for us to inspect.\n",
    "\n",
    "with open('sampleText.txt') as sampleText:\n",
    "    content = sampleText.read()\n",
    "    print(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that I didn't add a method for opening the file--it just defaulted to *read*. If we tried to edit this file, we'd get an error message because we didn't give the code permission to do that. Let's try opening it again with write permissions and writing some new content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In the first line, we've added a 'method' to our open function, which allows us to write to the file.\n",
    "\n",
    "with open('sampleText.txt', 'w') as sampleText:\n",
    "    newText = \"This is what happens if we open this file with 'w' permissions!\"\n",
    "    sampleText.write(newText)\n",
    "    \n",
    "#With this function, we'll re-read our file to see what it contains!\n",
    "with open('sampleText.txt','r') as sampleText:\n",
    "    newFileContent = sampleText.read()\n",
    "    print(newFileContent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice something? \n",
    "We overwrote the old contents of our sampleText.txt file with this new message! The old message is gone. (Need to see it for yourself? Use your file explorer to go open the text file in a text editor). If we want to *add* content without erasing what was already there, we can use the append method (`a`) instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('sampleText.txt', 'a') as sampleText:\n",
    "    newNewText = \"\\nAppend lets us add a new line without overwriting our last line!\"\n",
    "    sampleText.write(newNewText)\n",
    "    \n",
    "#With this function, we'll re-read our file to see what it contains!\n",
    "with open('sampleText.txt','r') as sampleText:\n",
    "    newFileContent = sampleText.read()\n",
    "    print(newFileContent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summary\n",
    "So far, we've covered the not-quite-as-basic way of handling files in Python. The file type is simple--Python works well with .txt files--but the `with open()` technique is a bit more advanced. In the next section of this Lab, we'll integrate these techniques to save and interact with data we retrieve from the web."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Automating downloading & API requests (and dealing with the results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains basic Python code to get you started downloading content from the web and from some APIs. Start by importing `requests`. This makes the `requests` package available for use in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "⭐ **During today's class, you will practice with one of the chunks of starter code below (A-E), based on your interests and project needs.** ⭐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Using Python to download HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Downloading the HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the URL between the single quotes with the URL for the site you'd like to download.\n",
    "url = \"https://sherlock-holm.es/stories/html/advs.html\"\n",
    "response = requests.get(url)\n",
    "response.encoding = 'utf-8'\n",
    "responseContent = response.text\n",
    "\n",
    "#Replace 'holmes.html' to name your file something that makes sense.\n",
    "with open('holmes.html', 'w') as output:\n",
    "    output.write(responseContent)\n",
    "    \n",
    "print(responseContent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What a mess! What you see is in the output from the cell above is *all* the HTML it took to render the Sherlocke Holmes story (or whatever website you entered). But I'm guessing you didn't really want all that mess. The next step is to *parse* the HTML you've retrieved from the web and, for that, we need another library, `BeautifulSoup`. We'll import that next. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parsing the HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will want to read the [documentation for Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#), but we can also try out a few things first before you dive in. First, let's *pretty print* that HTML to make it easier to read!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('holmes.html') as text:\n",
    "    soupA = BeautifulSoup(text, 'html.parser')\n",
    "print(soupA.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks a little more like *something*, but it's still got a lot of stuff in it that you might not be interested in. You'll likely want to target particular sections of the text and, if the HTML is structured well, you can use the tags to do exactly that! Let's start with the title of the page. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupA.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...but it still has tags around it. We can drop those by telling Python we're interested in the *text* or *string* contained between the tags, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupA.title.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupA.p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 Did that do what you expected? If not, that's because this method only gives you the *first* tag in the document. So, it worked, just not how you might've expected. \n",
    "\n",
    "To call *all* the paragraph-tagged text from the page, you'll need to use the `find_all()` function. This will return a list of all the things that mat your search. In our case, we'll tell it to look for `<p>` tags by passing 'p' into the function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pTags = soupA.find_all('p')\n",
    "print(pTags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, that's a bit messy. So, let's strip out the tags to make it easier to read (and run text analysis on!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in pTags:\n",
    "    print(p.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've got something workable, let's save that to a .txt file on your computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('justText.txt', 'a') as file:\n",
    "    for p in pTags:\n",
    "        file.write(p.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📂 If you've run these cells exactly once each, you will now have 2 files on your computer: one HTML file with the raw HTML output from the page we downloaded and one .txt file with just the paragraph text.\n",
    "\n",
    "#### Next steps\n",
    "BeautifulSoup is a very useful Python library, but it can take some getting used to. Using it here also requires knowledge of XML or a fair bit of persistent experimentation! We'll come back to it in a few modules--with the persistent experimentation approach. For those reasons, we'll stop the walkthrough here. \n",
    "\n",
    "Add cells to this notebook or start a new one and explore! Ideas to get you started:\n",
    "1. Copy my code and change the URL.\n",
    "    - See what happens when you re-run the cells with a different website. (Note: Sites with dynamic content will not work as well as sites with static content (e.g., page content that is generated based on user input vs. content that is written into the HTML).\n",
    "2. Examine the HTML of a website you're interested in scraping. (Use \"Inspect\" or \"View page source\" in your browser of choice).\n",
    "    - How is the HTML structured?\n",
    "    - What kinds of tags are used for the content that interests you? Are there `ids` or `attributes` assigned to items of interest? (Check out [W3Schools HTML resources](https://www.w3schools.com/html/) for a quick guide or refresher.\n",
    "3. Start with one thing you'd like to pull out of the HTML. \n",
    "\n",
    "If you get to a point where you can parse out the pieces you want (e.g., if all you really needed was the paragraph text), you might start thinking about how you will scale this effort. What would you need to do to scrape multiple pages? Based on your research interest and the structure of the pages you're scraping, how would you want to format your scraped text--as one document? As multiple documents? How would you do that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  B. Using Python to download a PDF\n",
    "In this section, you'll find starter code for downloading PDFs from a website. This is more straight-forward, in some ways, than the other methods we're looking at. But, in this approach, we won't open print the data from the PDF to this notebook--the encoding is a problem. Instead, we'll simply write the document to your computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://www.loc.gov/aba/publications/FreeLCSH/numerals.pdf\"\n",
    "    \n",
    "responseB = requests.get(url)\n",
    "    \n",
    "with open (\"numerals_2022.pdf\", 'wb') as pdf:\n",
    "    pdf.write(responseB.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next steps for this option would include exploring ways to download multiple PDFs (see Part 2 of this lab) and finding ways to extract the text from the PDFs and save it to a .txt file. Note: as of the writing of this lab, the example PDFs do not work with the most commonly recommended PDF library. If these are your project files, ask Megan for help. Read more about the process [here](https://automatetheboringstuff.com/2e/chapter15/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Using Python to download XML\n",
    "#### Downloading the XML\n",
    "We'll take the Library of Congress' *Chronicling America* API as our example for both XML and JSON. There are two reasons for that: 1) it provides well-structured examples of both data formats and 2) it highlights important differences in what API endpoints are designed for. \n",
    "\n",
    "If you're in a group that's looked at *Chronicling America* already, you may have discovered by now that the xml-encoded results don't include the full text of the articles; instead, it supplies *metadata*. Fortunately, the JSON endpoint *does* include full text. So, if you're not interested in the full text of this specific resource, skip down to section D. If you're interested in XML, keep reading in this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the URL between the single quotes with the URL for the site you'd like to download.\n",
    "urlC = \"https://chroniclingamerica.loc.gov/search/pages/results/?andtext=wisconsin&format=atom\"\n",
    "\n",
    "responseC = requests.get(urlC)\n",
    "with open('chronicling.xml', 'w') as file:\n",
    "    file.write(responseC.text)\n",
    "    \n",
    "print(responseC.text)\n",
    "responseC_XML = responseC.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parsing the XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will want to read the [documentation for Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#), but we can also try out a few things first before you dive in. First, let's *pretty print* that HTML to make it easier to read!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupC = BeautifulSoup(responseC_XML, 'xml')\n",
    "print(soupC.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks a little more like *something*, but it's still got a lot of stuff in it that you might not be interested in. You'll likely want to target particular sections of the text and, if the XML is structured well, you can use the tags to do exactly that! Since this is a list of search results, let's experiment with the entry tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupC.entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soupC.find_all('entry')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, that's a bit hard to read. So, let's pull out just the text for each entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titlesC = soupC.find_all('entry')\n",
    "for entry in titlesC:\n",
    "    print(entry.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's imagine for a minute that you'd like to systematically look at each of these entries in closer detail. We could use `BeautifulSoup` to create a .txt file containing the links to the landing page for each entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('justURLs.txt', 'a') as file:\n",
    "    entryLinks = soupC.find_all('link')\n",
    "    for link in entryLinks:\n",
    "        file.write(link.get('href'))\n",
    "        file.write('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📂 If you've run these cells exactly once each, you will now have 2 files on your computer: one XML file with the raw XML output from the page we downloaded and one .txt file with just the links. \n",
    "\n",
    "For next steps, I'll ask you to spend time thinking about what you're retrieving from the API, how to access key items of interest, and how you might go about automatically retrieving the full result set. (Remember, we only looked at page 1 of many).\n",
    "\n",
    "#### Next steps\n",
    "BeautifulSoup is a very useful Python library, but it can take some getting used to. Using it here also requires knowledge of XML or a fair bit of persistent experimentation! We'll come back to it in a few modules--with the persistent experimentation approach. For those reasons, we'll stop the walkthrough here. \n",
    "\n",
    "Add cells to this notebook or start a new one and explore! Ideas to get you started:\n",
    "1. **If you haven't, explore the URL structure used by your API. How can you manipulate the API to create a query you're interested in?**\n",
    "2. **Copy my code and change the query in the URL.**\n",
    "    - Sticking with the example for now, see what happens when you re-run the cells with a different query. \n",
    "3. **Examine the XML response from the API.**\n",
    "    - How is the XML structured? Are there nested values? Does the structure capture relationships you're interested in?\n",
    "    - What kinds of tags are used for the content that interests you? Are there `ids` or `attributes` assigned to items of interest? (Check out [W3Schools XML resources](https://www.w3schools.com/xml/) for a quick guide or refresher.\n",
    "4. **Start with one thing you'd like to pull out of the XML--can you get Python to return just that item?**\n",
    "    - From there, you might think about whether you want a .txt file (if you're looking at full text) or another data format, such as .csv, which you might use to visualize patterns in metadata. You may also wish to preserve the original XML, but we've already done that!\n",
    "5. **Copy my code and try it out with an API of your choice.** (Note: You will need to change the query to match the structure used by your API. You may also need to consider changing the tags we used to parse the sample, but that will depend on how the responses from your API are organized)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## D. Using Python to download JSON\n",
    "\n",
    "In this section, you'll find starter code for sending an API query and working with JSON output. To save the JSON, we'll use a python package called `json` (clever, huh?). First, we'll import it, then we'll use it when we go to save the data we've retrieved from the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's send our API request and print the result to screen. `requests` has a built-in JSON function (different from the library we just imported), so we'll use that just to get a preview of our data. Note the information at the start of the response: there are 1,870,346 items that match our query and this *page* of the results displays just 20 of them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the URL in the quotation marks below with the URL for your API request\n",
    "preview = requests.get(\"https://chroniclingamerica.loc.gov/search/pages/results/?andtext=wisconsin&format=json\").json()\n",
    "preview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our preview just printed the results to the screen, but you will want to save your data for future project work. Run the code below to write the data to a .json file on your computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the file name in the quotation marks with a file name that's meaningful in your context.\n",
    "with open(\"chroniclingSearch.json\", \"w\") as write_file:\n",
    "    json.dump(preview, write_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing the JSON dictionary\n",
    "Now that we have our json and have saved a copy, there are many ways to work with it. \n",
    "\n",
    "With json results in particular, learning the dictionary structure in Python is important. Without a working comfort or familiarity with this structure, you won't be able to efficiently access or work with the data. \n",
    "\n",
    "Let's say we just want to peek at the OCR text in the json results we've gotten so far. The cell below will print the OCR text of each result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In the lines below, we're looping over the dictionary and calling items in the dictionary with their \"key\". \n",
    "#First, we're asking for the content of 'items', which is itself contains multiple entries.\n",
    "#Then, we're asking for the entry with an index of i. (Remember, the index is the item's place in the sequence).\n",
    "#And last, from the entry in the i position in the list, we're asking just for the value stored in ocr_eng.\n",
    "i = 0\n",
    "for result in preview:\n",
    "    print(preview['items'][i]['ocr_eng'])\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted this data, as is, you could write it to a text file. But, I suspect you might actually want to save the text in separate files, one for each result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 0\n",
    "for result in preview:\n",
    "    with open('%s.txt' %(i), 'w',encoding='utf-8')  as file:\n",
    "        file.write(preview['items'][i]['ocr_eng'])\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📂 If you've run these cells exactly once each, you will now have 6 files on your computer: one JSON file with the raw JSON output from the page we downloaded and 5 .txt files with just the OCR text.\n",
    "\n",
    "### Next steps\n",
    "A logical next step for working with JSON data would be to convert it to a CSV file, which is more commonly used with the software we'll use. One approach would be to use the `pandas` library, which we'll look at later in this class. For now, pause on the format conversion and spend time thinking about what you're retrieving from the API, how to access key items of interest, and how you might go about automatically retrieving the full result set. (Remember, we only looked at page 1 of many).\n",
    "\n",
    "1. **If you haven't, explore the URL structure used by your API. How can you manipulate the API to create a query you're interested in?**\n",
    "2. **Copy my code and change the query in the URL.**\n",
    "    - Sticking with the example for now, see what happens when you re-run the cells with a different query.  \n",
    "3. **Examine the JSON response from the API.**\n",
    "    - How is the JSON structured? Are there nested values? Does the structure capture relationships you're interested in?\n",
    "    - What kinds of tags are used for the content that interests you? Are there ids or attributes assigned to items of interest? (Check out [W3Schools JSON resources](https://www.w3schools.com/js/js_json_intro.asp) for a quick guide or refresher.\n",
    "4. **Start with one thing you'd like to pull out of the JSON--can you get Python to return just that item?**\n",
    "From there, you might think about whether you want a .txt file (if you're looking at full text) or another data format, such as .csv, which you might use to visualize patterns in metadata. You may also wish to preserve the original JSON, but we've already done that!\n",
    "5. **Copy my code and try it out with an API of your choice.** (Note: You will need to change the query to match the structure used by your API. You may also need to consider changing the tags we used to parse the sample, but that will depend on how the responses from your API are organized)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### E. Downloading Tweets with `twarc`\n",
    "\n",
    "If you're interested in working with Tweets, you're in luck!* DocNow has created a command line tool and python library for you AND released an update for the new API. You'll run it from the command line instead of Jupyter Notebook. Check it out at: https://twarc-project.readthedocs.io/en/latest/twarc2_en_us/\n",
    "\n",
    "\\*Well, sort of, assuming the free tier of the Twitter API remains available for educational and research purposes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ⚬ An aside on object-oriented programming and dot notation\n",
    "This point is more fundamental than it is basic in the sense that it's central to how Python works as a programming language, but it's a more abstract concept that may not completely make sense right away. AND! You don't absolulely need this point to make sense to start writing Python. So, I stuck it at the end!\n",
    "\n",
    "`requests.get()` is an example of dot notation in Python. Dot notation is a way of telling your computer where to find what you want or where to do the thing you're asking it to do. \n",
    "\n",
    "You may recall that I referred to Python as an *object-oriented programming language*. This means that Python treats basically everything as an *object* with a set of attributes and actions that go with it. In this case, `requests` is our object and `get()` is a method (function) that is in the `requests` package. So, the dot notation tells your computer where to find the `get()` function.\n",
    "\n",
    "You can also use dot notation to tell Python to perform an action *on* an object. For example, if you want to add something to a list you've very creatively called 'myList', you would use `myList.append('something')`. In this case, you're telling the computer *where* to add that thing you want to add."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Automating the retrieval of multiple things\n",
    "In this section, I'll share an example that uses basic Python concepts to automate the downloading of multiple things. We'll download multiple PDFs from the same website. \n",
    "\n",
    "In Part 2B, we have a [website with 27 PDFs listing Library of Congress Subject Headings](https://www.loc.gov/aba/publications/FreeLCSH/freelcsh.html#Individual) for each edition. There are 10 editions available online--that's a lot of manual downloading. Instead, we can study the URL, determine the pattern, then use that pattern to automatically download all the files.\n",
    "\n",
    "The first thing I noticed is that the files are named for the first letter of the subject heading: \"numerals.pdf\", \"a.pdf\", \"b.pdf\",  etc. If we create a list of file names, we can use a strange little trick in Python to change the letter in the URL. That strange trick is called *string formatting* and it's a bit unlike other conventions in Python. \n",
    "\n",
    "### An aside on string formatting 🧶\n",
    "In the URL below, you'll notice a %s and after the URL, you'll notice a `%(fileName)`. That's the string formatting syntax. It basically says \"hey computer, where you see the `%s`, treat it like a placeholder. When you see `%` again, insert whatever's inside the parentheses.\" \n",
    "\n",
    "You might think of the `%` as something like an asterisk or a footnote when you're reading text--as a reader familiar with this convention, you know you need to look for that asterisk or footnote character elsewhere on the page for more information.\n",
    "\n",
    "In our case, we're using a variable that will change with each iteration through our list. It will start with \"NUMERALS\", then move on to \"A\", then \"B\", then C\". If that doesn't make sense, that's okay. It's more advanced than what we've done so far, but I include it because it will help you move from \"baconeggs\" to doing something practical with Python. The script below will take some tweaking for use in other contexts, but I can help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This is our list of file names. We'll ask Python to plug these values into the URL below, one by one, to download each file. \n",
    "# We have to set this up first so that when we refer to it, the computer knows what we're talking about.\n",
    "fileNameList = ['NUMERALS','A','B','C']\n",
    "\n",
    "#This is a loop that lets us do the same task over and over for each of the files we want to download. \n",
    "#The loop will go over every item in the list\n",
    "for fileName in fileNameList:\n",
    "    #This is where we give the URL for the file we want to download. You'll see the funky string formatting in this line.\n",
    "    url = \"https://www.loc.gov/aba/publications/FreeLCSH/%s.pdf\" %(fileName)\n",
    "    \n",
    "    #In this line, we're \"getting\" the file and storing that information in a variable\n",
    "    responseB = requests.get(url)\n",
    "    \n",
    "    #In this line, we're writing the file to your computer\n",
    "    with open (\"%s_2022.pdf\" %(fileName), 'wb') as pdf:\n",
    "        pdf.write(responseB.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Reflection\n",
    "If you've made it this far, please take a few minutes to share your progress so far. Add a Markdown cell below and tell me what you tried, what happened, and what you make of it all."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}