Introduction to Web scraping<\/p>\n
Web scraping or crawling is the act of fetching data from a third party website by downloading and parsing the HTML code to extract the data you want. It can be done manually, but generally this term refers to the automated process of downloading the HTML content of a page, parsing\/extracting the data, and saving it into a database for further analysis or use.<\/p>\n
Web fundamentals<\/p>\n
The internet is really complex : there are many underlying techologies and concepts involved to view a simple web page in your browser. I don\u2019t have the pretention to explain everything, but I will show you the most important things you have to understand to extract data from the web.<\/p>\n
HyperText Transfer Protocol<\/p>\n
From Wikipedia :<\/p>\n
The Hypertext Transfer Protocol (HTTP) \u00a0is an\u00a0application layer<\/a>\u00a0protocol in the\u00a0Internet protocol suite<\/a>\u00a0model for distributed, collaborative,\u00a0hypermedia<\/a> information systems. HTTP is the foundation of data communication for the\u00a0World Wide Web<\/a>, where\u00a0hypertext<\/a>\u00a0documents include\u00a0hyperlinks<\/a>\u00a0to other resources that the user can easily access, for example by a\u00a0mouse<\/a>\u00a0click or by tapping the screen in a web browser. Hypertext is structured text that uses logical links (hyperlinks) between nodes containing text. HTTP is the protocol to exchange or transfer hypertext.<\/p>\n So basically, as in many network protocols, HTTP uses a client\/server model, where an HTTP client (A browser, your Java program, curl, wget\u2026) opens a connection and sends a message (\u201cI want to see that page : \/product\u201d)to an HTTP server (Nginx, Apache\u2026). Then the server answers with a response (The HTML code for exemple) and closes the connection. HTTP is called a stateless protocol, because each transaction (request\/response) is independant. FTP for example, is stateful.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":"